I have a service on Debian 9. I also have a bash restarter so if the service shuts down unexpectedly, the restarter will make it run again.
The restarter basically does:
if (this service is not running); then
run service
fi
The problem I have is that sometimes, after the service shuts down unexpectedly, the restarter makes it run again but I get the error:
failed to bind socket acceptor
The service is running but not really working.
What can I add on my restarter so it checks the socket is available first and then, run the service? I am trying a lot of things posted here but nothing works so far.
Your problem is probably caused by a connection in TIME_WAIT state (see e.g. here).
You can avoid this problem by using SO_REUSEADDR in the application.
If you can not alter the application you would have to check for this condition in the restarter and delay service start until the TIME_WAIT connection vanishes.
This check can be done by analyzing the output of netstat, e.g.:
while netstat -n | egrep ':5678.*:.*TIME_WAIT' >/dev/null ; do sleep 0.1 ; done
Where 5678 is your service port number.
Good luck!
EDIT> Restarter snippet with a check:
pidof service >/dev/null
PID1=$?
if [ $PID1 -eq 1 ] ; then
while netstat -n | egrep ':5678.*:.*TIME_WAIT' >/dev/null ; do sleep 1 ; done
screen -A -dmS service1 gdb --batch -x /home/server/crashreport.gdb /home/server/bin/service
fi
Note that a shorter form is possible:
if ! pidof service >/dev/null ; then
while netstat -n | egrep ':5678.*:.*TIME_WAIT' >/dev/null ; do sleep 1 ; done
screen -A -dmS service1 gdb --batch -x /home/server/crashreport.gdb /home/server/bin/service
fi
If your restarter runs this check in a loop you should give screen and gdb its time to start the service (otherwise the service could be run two times which might cause a similar error message as the one in your question...)
Related
I want to check if tomcat server is really started up. When you start tomcat, you get an entry like "Server startup" in catalina.out. Once I got this, the script should go ahead.
That's my code
echo "Waiting for Tomcat"
if [ $(tail -f /home/0511/myapp/logs/catalina.out | grep "Server startup" | wc -l) -eq 1 ]; then
echo "Tomcat started"
fi
#further code...
Output:
Waiting for Tomcat
|
So, I am sure, after 30-60 seconds, the "tail .. | wc -l" gets 1. However, I cannot match it with my code above. Nothing happens, I have to interrupt per CTRL C.
What is wrong or is there any better solution for my intention?
Try this:
while true;do
if tail -n 100 /home/0511/myapp/logs/catalina.out | grep -q "Server
startup";then
echo "Tomcat started"
exit 0
fi
done
So you constantly check the last 100 lines from the log, and if match, exit with a message.
Another (more elegant) solution without a loop:
if tail -f /home/0511/myapp/logs/catalina.out | grep -q "Server
startup";then
echo "Tomcat started"
fi
You said "I want to check if tomcat server is really started up". If you check the log with tail and grep, in worst case scenario, you could detect an old start which ended with a crash.
Tomcat server when is started it is listening to a certain port(e.g. 8080). So you should check if tomcat server is listening on that port.
If you are using a different port replace 8080 in following lines with your custom port.
In order to display tomcat status you should use netstat. Example of a line returned by netstat:
TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING
In order to display only if Tomcat is started or not, you could use:
netstat -an|grep -e ":8080[^0-9].*[0-9].*LISTENING" && echo "Tomcat started" || echo "Tomcat not started"
The grep expression matches ":" followed by port 8080, followed by non-digit, followed by any characters, followed by digit, followed by any characters, followed by "LISTENING".
In order to wait for Tomcat you could use:
echo Waiting for Tomcat port to be ...
until netstat -an | grep -e ":8080[^0-9].*[0-9].*LISTENING" > /dev/null ; do
sleep 1 ;
done
echo Tomcat started
grep doesn't exit as soon as it found a match; it keeps reading for further matches.
To make grep not produce any output, but instead exit with 0 status as soon as a match is found, use the -q option:
if tail -f /home/0511/myapp/logs/catalina.out | grep -q "Server startup"; then
That said, a log message isn't the most reliable way to check if a server is actually up and serving. Instead, you could try, for example, repeatedly polling it with curl instead.
I have a script to start and fork a netcat process. After a while, the netcat process stops logging output. Remote computers are supposed to connect to the socket and send a message every few hours, but it seems like the netcat process dies/halts after a while, because there's usually only one message from the remotes on the hour after the daemon starts, and then no more follow. I ensured it wasn't a problem with the remotes not sending their information to the socket; so it seems like something to do with the netcat process dying out. When I run atop, the process is still alive, but if I try to connect to the socket manually and send something, it doesn't log it to the output file.
dstart(){
if [ -f /run/mynetcat.pid ]; then
echo "Netcat instance running on "$(cat /run/mynetcat.pid)
exit 1
else
echo "Starting Netcat instance"
mkdir -p /var/log/mynetcat/
(setsid nc -l -k -p 25001 >> /var/log/mynetcat/mynetcat.log 2> /var/log/mynetcat/mynetcat.err & echo $! > /run/mynetcat.pid)&
return 0
fi
}
###later on in the script
case "$1" in
start)
dstart
;;
new here have learned tons over the years and happen to have a pretty puzzling question. I just purchased a product to help edit photos in Adobe Lightroom. Its called a Loupedeck. Comes with software and is a device that controls all the parameters in Lightroom (makes editing way quicker)
Once I installed their loupdeck software and opened Lightroom I immediately got this prompt.
"Loupedeck needs access to tcp ports 23515 and 23516.
Other process is currently occupying 23515.
Loupedeck will not work until that application is closed."
I have been searching for days trying to find what app I have installed thats using or used port 23515. I have tried checking in terminal with the simple netstat command, and found nothing. I've used little snitch to see if any programs are using that port, still nothing. Im a little lost and sadly the company is also lost and has no advice.
Computer - Macbook pro 2011 17" 2.3ghz OSX 10.12.6
Here is the error in Lightroom
From this answer:
lsof -n -i4TCP:23515
lsof -n -i TCP:23515 | grep LISTEN
lsof -n -i:23515 | grep LISTEN
Netstat may also work:
netstat -ap tcp | grep -i "23515"
Any of these commands should give you the name and PID of the process.
Once you have the PID, you can kill the process, or if you know the process name is process_name you can use killall -KILL process_name .
Just for the record though, your terminal isn't running an administrative shell by default. In order to execute commands in terminal as an administrator, you have to use the command sudo su and then enter your password. Once you have a prompt with a #, you can try these commands again, just to be sure.
In one command find and kill:
kill -9 `lsof -i : 23515 -t` or if the existing process was launched by root or an other user: sudo kill -9 `lsof -i : 23515 -t`
lsof -i : 23515 returns the process id using the port 23515 on your machine.
You may also want to restart your computer, just in case the process using this port went into a "zombie" mode or lock the port and went off without freeing the port.
You can try to fine the process listening to those ports:
lsof -i :23515 -t
If you want to save time in the long run, you can add this script "kill-process.sh" into your /usr/local/bin
#!/bin/bash
echo Killing process..
usage() { echo "Usage: $0 [-p <port-number>]" 1>&2; exit 1; }
while getopts ":p:" o; do
case "${o}" in
p)
p=${OPTARG}
((p > 0 )) || usage
;;
*)
usage
;;
esac
done
shift $((OPTIND-1))
if [ -z "${p}" ]; then
usage
fi
PID=$(lsof -ti:$p)
if [ -z "${PID}" ]; then
echo "No process running on port ${p}"
else
kill $PID
echo "Killed process PID: ${PID}"
fi
Add this line into your .zshrc
alias kill-process="sh /usr/local/bin/kill-process.sh"
Then just run from your terminal
kill-process -p 23515
You will save time in the long run and this is useful if you need to close any process in any port by specifying it as a parameter.
I'm trying to do a custom dockerfile with jenkins on it. I would to wait until port 8080 is open instead of doing an ugly 'sleep 60' with netcat but I'm not very confident with bash scripts and netcat.
Here is an example of what i'm trying to do:
#!/bin/bash
opened=0
while [ "$opened" == "0" ]; do
echo "Waiting jenkins to launch on 8080..."
nc -vz localhost 8080
done
echo "Jenkins launched"
You can't set netcat to wait until some port is open, so you have to add part for waiting before next check is made. Try this:
#!/bin/bash
echo "Waiting jenkins to launch on 8080..."
while ! nc -z localhost 8080; do
sleep 0.1 # wait for 1/10 of the second before check again
done
echo "Jenkins launched"
I suggest the following one liners:
## netcat version:
timeout 22 sh -c 'until nc -z $0 $1; do sleep 1; done' stackoverflow.com 443
## pure bash version:
timeout 22 bash -c 'until printf "" 2>>/dev/null >>/dev/tcp/$0/$1; do sleep 1; done' stackoverflow.com 443
Both commands exit as soon as connection is established, trying every second for up to 22 seconds.
Note that thanks to timeout command exit code is 0 when port is accessible otherwise 124 (if no connection established within given time).
As suggested here, you could also do the following if you don't have nc installed but just bash and coreutils:
#!/bin/bash
echo "Waiting jenkins to launch on 8080..."
while ! timeout 1 bash -c "echo > /dev/tcp/localhost/8080"; do
sleep 1
done
echo "Jenkins launched"
I have found this a common enough problem to write a utility to wait for a port to open, with an optional timeout:
# without timeout
wait-port localhost:8080
# timeout after a minute
wait-port -t 60000 localhost:8080
It's open source and available at github.com/dwmkerr/wait-port. Hopefully others will find it useful!
To expand on user987339's answer, here's how to easily wait for a port in your terminal:
waitport function
Add this function to your ~/.bashrc setup file:
waitport() {
while ! nc -z localhost $1 ; do sleep 1 ; done
}
Log out then back in to load ~/.bashrc. Then, run this command to verify that port 3000 has a server listening to it:
$ waitport 3000
Connection to localhost port 3000 [tcp/hbci] succeeded!
This has been validated on macOS. It might not work on Fedora/CentOS, as they lack the -z option for netcat.
To add onto the excellent answers above, if this is something used very often it may be worthwhile to use tooling for that purpose. I wrote and use uup all the time for this use case.
In your example, the command to run would be:
uup localhost:8080 -r
providing an output like:
Here is a for-loop example that has a timeout, so it tries e.g. for 10 times, with exponential backoff (2,4,8,16 seconds etc), but finally gives up. Netcat has also 1 second timeout.
for EXPONENTIAL_BACKOFF in {1..10}; do
nc -w 1 -z db.local 3306 && break;
DELAY=$((2**$EXPONENTIAL_BACKOFF))
echo "db not yet available, sleeping for $DELAY seconds"
sleep $DELAY
done
The output is:
db not yet available, sleeping for 2 seconds
db not yet available, sleeping for 4 seconds
db not yet available, sleeping for 8 seconds
db not yet available, sleeping for 16 seconds
I use this script to check the port before running tests on CI.
#!/bin/bash
for _ in `seq 1 20`; do
echo -n .
if nc -z localhost $1; then
exit 0
fi
sleep 0.5
done
exit 1
$ bin/wait-port 3306
Here is my one-line Bash solution (with netcat) that waits for 10 sec for a TCP connection, and give you feedback whether succeeded or not and while is waiting, and return an exit 0 code if the port is open, otherwise 1:
bash -c 'echo -n "Waiting port 8080 .."; for _ in `seq 1 40`; do echo -n .; sleep 0.25; nc -z localhost 8080 && echo " Open." && exit ; done; echo " Timeout!" >&2; exit 1'
You can replace the hardcoded port 8080 by $1 and remove the bash -c if the snippet is saved in a script file wait-port than then is called within a console with: wait-port 8080.
This is a recording of 3 terminals, two waiting until a port is opened and the other terminals open one of the port, so while one of the wait succeed, the other timed-out:
Although the line has many instructions not one, it may be useful if you need to execute the wait "remotely" in a host where you cannot store the script first, e.g. in a Docker container.
I used this to wait for a couple of ports to be open, without netcat:
while (! (: </dev/tcp/localhost/27017) &> /dev/null || ! (: </dev/tcp/localhost/9200) &> /dev/null); do
sleep 2;
done
Change localhost and the ports as needed.
For those people who are having trouble with nc: invalid option -- 'z'
I was trying to set this up in a docker image. Surprisingly, there was no option of -z in nc in that image.
Image was - Linux elasticsearch 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
I used the following loop to wait until the port was opened.
#!/bin/bash
echo "Waiting elastic search to launch on 9200..."
open=0;
while [ $open -eq 0 ]
do
check_port=`nc -v -w 1 -i 1 127.0.0.1 9200 &> /dev/stdout`
echo $check_port
if [[ "$check_port" == *"Connected to"* ]]
then
break
fi
sleep 1
done
echo "Elastic Search launched"
Following is the one-liner of the above script:
open=0;while [ $open -eq 0 ]; do check_port=`nc -v -w 1 -i 1 127.0.0.1 9200 &> /dev/stdout`; echo $check_port; if [[ "$check_port" == *"Connected to"* ]]; then break; fi; sleep 1; done
I have written a utility to wait for a port to open, it can also check MySQL, PostgreSQL, Redis and etc availability.
# Checking TCP port
wait4x tcp localhost:8080
# Checking TCP port with specific timeout (5 Minutes)
wait4x tcp localhost:8080 -t 5m
It's open source and available at https://github.com/atkrad/wait4x. Hopefully others will find it useful!
If you are looking for a docker-run-one-liner, check out my Docker image based on the netcat loop:
$ docker run --rm igops/wait-for-port HOST PORT
E.g.,
$ docker run --rm igops/wait-for-port 172.17.0.1 80
will query 172.17.0.1:80 each 0.1s and exit when the connection is established:
Waiting for 172.17.0.1:80...
OK
Waiting for another container to response:
$ docker network create my-bridge
$ docker run --rm -d --net my-bridge --net-alias my-mongo mongo
$ docker run --rm --net my-bridge igops/wait-for-port my-mongo 27017
$ echo "Mongo is up at this point, do some useful stuff here"
Waiting for some service on the docker host:
$ docker run --rm --add-host="docker-host:host-gateway" igops/wait-for-port docker-host 22
$ echo "SSH server is running"
Waiting for another container which published some port:
$ docker run --rm -d -p 27017:27107 mongo
$ docker run --rm --add-host="docker-host:host-gateway" igops/wait-for-port docker-host 27017
$ echo "Mongo is up"
Read more
I have a bash shell script which does a bunch of stuff before trying to mongorestore.
I want to make sure that not only MongoDB is up, but it is also ready to accept connections before i try restore.
Right now, what I am seeing is, mongo process is up but it take 45+ seconds to do the initial setup (setting up journal files etc) before it is ready to accept connections. Ideally I want to keep testing the connection in a loop and only when I am able to connect, I want to run mongorestore.
Can someone show me how to do this in Bash or point me in the right direction?
To test the connection in a loop like you suggest,
until nc -z localhost 27017
do
sleep 1
done
A solution using MongoDB Tools. Useful in a docker container or something similiar where you do not want to install nc.
until mongo --eval "print(\"waited for connection\")"
do
sleep 60
done
Based on that other guy's answer.
I recently had the same problem. I decided to configure mongod to log all it's output to a logfile and then wait in a loop checking the logfile until we see some output that suggests mongod is ready.
This is an example logfile output line we need to wait for:
Tue Dec 3 14:25:28.217 [initandlisten] waiting for connections on port 27017
This is the bash script I came up with:
#!/bin/bash
# Initialize a mongo data folder and logfile
mkdir -p /data/db
touch /var/log/mongodb.log
# Start mongodb with logging
# --logpath Without this mongod will output all log information to the standard output.
# --logappend Ensure mongod appends new entries to the end of the logfile. We create it first so that the below tail always finds something
/usr/bin/mongod --quiet --logpath /var/log/mongodb.log --logappend &
# Wait until mongo logs that it's ready (or timeout after 60s)
COUNTER=0
grep -q 'waiting for connections on port' /var/log/mongodb.log
while [[ $? -ne 0 && $COUNTER -lt 60 ]] ; do
sleep 2
let COUNTER+=2
echo "Waiting for mongo to initialize... ($COUNTER seconds so far)"
grep -q 'waiting for connections on port' /var/log/mongodb.log
done
# Now we know mongo is ready and can continue with other commands
...
Notice the script will not wait forever, it will timeout after 60s - you may or may not want that depending on your use case.
I needed Mongo running in Docker to initialize before creating a user. I combined the answers of Tom and Björn. This is the script I am using:
#!/bin/bash
# Wait until Mongo is ready to accept connections, exit if this does not happen within 30 seconds
COUNTER=0
until mongo --host ${MONGO_HOST} --eval "printjson(db.serverStatus())"
do
sleep 1
COUNTER=$((COUNTER+1))
if [[ ${COUNTER} -eq 30 ]]; then
echo "MongoDB did not initialize within 30 seconds, exiting"
exit 2
fi
echo "Waiting for MongoDB to initialize... ${COUNTER}/30"
done
# Connect to the MongoDB and execute the create users script
mongo ${FWRD_API_DB} /root/create-user.js --host ${MONGO_HOST} -u ${MONGO_ROOT_USER} -p ${MONGO_ROOT_PASSWORD} --authenticationDatabase admin
While Tom's answer will work quite well in most situations, it can fail if you are running your script with the -e flag. I mean you run your script with set -e at the very top. Why? Because Tom's answer relies on the exit status of the previous command, in this case grep -q which will fail if it does not find the required string, and therefore the entire script will fail. The -e flag documentation gives a clue on how to avoid this problem:
The shell does not exit if the command that fails is part of the
command list immediately following a while or until keyword, part of
the test in an if statement, part of any command executed in a && or
|| list except the command following the final && or ||, any command
in a pipeline but the last, or if the command’s return status is being
inverted with !.
So, one solution is to make the grep command part of the while condition. However, since he is running mongodb with the --logappend option the search string could appear as a result of a previous run. I combined that other guy answer with Tom's answer and it works really well:
# Wait until mongo logs that it's ready (or timeout after 60s)
COUNTER=0
while !(nc -z localhost 27017) && [[ $COUNTER -lt 60 ]] ; do
sleep 2
let COUNTER+=2
echo "Waiting for mongo to initialize... ($COUNTER seconds so far)"
done
I find that using tomcat is the best solution because it actually tests if there is something listening.
Possible Docker solution:
Given the docker_id for me it works reading the docker logs like:
until [ $(docker logs --tail all $docker_id | grep "waiting for connections on port" | wc -l) -gt 0 ]; do
printf '.'
sleep 1
done
then continue with any mongo-dependent task.
This approach is using in bitnami mongodb helm chart, but requires installed mongodb shell client:
mongosh --host {your_mongo_host} --port {your_mongo_port} --eval "db.adminCommand('ping')"