I'm trying to set up a script to execute tests for my node.js program, which uses MongoDB. The idea is that I want to have a script I can run that:
Starts the MongoDB process, forked as a daemon
Pre populates the database with some test data
Starts my node server with forever, so it runs as a daemon
Run my tests
Drop the test data from the database
I have a crude script that performs all these steps. My problem is that MongoDB takes a variable amount of time to set up, which results in sleep calls in my script. Consequently it only works occasionally.
# the directory of this script
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# launch mongodb.
$DIR/../../db/mongod --fork --logpath ./logs/mongodb.log --logappend --dbpath ./testdb/ --quiet
# takes a bit of time for the database to get set up,
# and since we make it a daemon process we cant run the tests immediately
sleep 1
# avoid EADDRINUSE errors because existing node servers are up.
killall node &> /dev/null
# start up our node server using a test database.
forever start $DIR/../main.js --dbname=testdb --logpath=test/logs/testlog.log
# takes a bit of time for node to get set up,
# and since we make it a daemon process we cant run the tests immediately
sleep 1
# run any database setup code (inject data for testing)
$DIR/../../db/mongo 127.0.0.1:27017/testdb $DIR/setup.js --quiet
# actually run the tests
node $DIR/tests.js
# kill the servers (this could be a little less heavy handed...)
killall node &> /dev/null
killall forever &> /dev/null
# finally tear down the database (drop anything we've added to the test db)
$DIR/../../db/mongo 127.0.0.1:27017/testdb $DIR/teardown.js --quiet
# and then shut mogodb down
kill -2 `ps ax | grep mongod | grep -v grep | awk '{print $1}'`
What is the best way to go about what I'm trying to do? Am I going down a rabbit hole here, or am I missing something in the MongoDB docs?
Ask yourself what your purpose of your testing is: Is it to test the actual DB connection in your code, or focus on whether your code handles and processes data from the DB correctly?
Assuming your code is in Javascript, if you strictly want to test that your code logic is handling data correctly, and are using a MongoDB wrapper object class (i.e. Mongoose), one thing you may be interested to add to your workflow is the creation and running of spec tests using the Jasmine test suite.
This would involve writing test-code, mocking-up test data as javascript objects. Yes, that means any actual data from the DB itself will not be involved in your spec tests. Since after all, your primary purpose is to test if your code is logically working right? It's your project, only you know the answer :)
If your main problem is how to find out when mongod actually starts why don't you write a script which will tell you that ?
For example you can write a until loop and check if the client can connect properly on the mongo server, based on the return value. For example instead of using sleep 1 use something like that:
isMongoRunning=-1
until [[ ${isMongoRunning} -eq 0 ]]; do
sleep 1
$DIR/../../db/mongo 127.0.0.1:27017/testdb $DIR/empty.js --quiet
isMongoRunning=${?}
done
This loop will end only after mongodb start.
Also of you would like to improve the stopping of mongodb add --pidfilepath to your mongod execution line so you can easily find which process to terminate.
Related
So I need to create a Docker Image of cassandra that have keyspace. So i made this docker file
FROM cassandra
WORKDIR /usr/app
COPY script.cql ./script.cql
COPY entrypoint-wrap.sh ./entrypoint-wrap.sh
CMD ["cassandra", "-f"]
RUN bash entrypoint-wrap.sh
and script.cql contains
CREATE KEYSPACE project WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 3};
and i want to run this script
#!/bin/bash
echo "checkcqlsh"
program=no
while [ $program = no ]
do
if ! [ -x "$(command -v cqlsh)" ]; then
echo 'Error: cqlsh is not included.'
program=no
else
program=yes
cqlsh -f script.cql
fi
done
But when it check if condition the image building stops because of an error that DB didn't have started yet. How to check if the cassandra DB is up and running?
Having CMD and then RUN will not work as they are executed at different lifecycles.
One ways is to have a custom entrypoint, that will start Cassandra in the background, wait for it's getting online, apply the schema change, and wait indefinitely, or until SIGTERM (waiting for a signal is better, as it allows to shutdown Cassandra clearly).
Another way - pre-initialize the container during build time, something like this:
Start Cassandra in background with RUN
Wait for it's started, and apply schema changes
Terminate Cassandra
then, just normal cassandra -f will work as CMD command.
Example of such thing is available in the DataStax's JIRA for Java driver, and has link to a gist that have Dockerfile + scripts.
I'm trying to write a bash script.
The script should check if the MC server is running. If it crashed or stopped it will start the server automatically.
I'll use crontab to run the script every minute. I think I can run it every second it won't stress the CPU too much. I also would like to know when was the server restarted. So I'm going to print the date to the "RestartLog" file.
This is what I have so far:
#!/bin/sh
ps auxw | grep start.sh | grep -v grep > /dev/null
if [ $? != 0 ]
then
cd /home/minecraft/minecraft/ && ./start.sh && echo "Server restarted on: $(date)" >> /home/minecraft/minecraft/RestartLog.txt > /dev/null
fi
I'm just started learning Bash and I'm not sure if this is the right way to do it.
The use of cron is possible, there are other (better) solutions (monit, supervisord etc.). But that is not the question; you asked for "the right way". The right way is difficult to define, but understanding the limits and problems in your code may help you.
Executing with normal cron will happen at most once per minute. That means that you minecraft server may be down 59 seconds before it is restarted.
#!/bin/sh
You should have the #! at the beginning of the line. Don't know if this is a cut/paste problem, but it is rather important. Also, you might want to use #!/bin/bash instead of #!/bin/sh to actually use bash.
ps auxw | grep start.sh | grep -v grep > /dev/null
Some may suggest to use ps -ef but that is a question of taste. You may even use ps -ef | grep [s]tart.sh to prevent using the second grep. The main problem however with this line is that that you are parsing the process-list for a fairly generic start.sh. This may be OK if you have a dedicated server for this, but if there are more users on the server, you run the risk that someone else runs a start.sh for something completely different.
if [ $? != 0 ]
then
There was already a comment about the use of $? and clean code.
cd /home/minecraft/minecraft/ && ./start.sh && echo "Server restarted on: $(date)" >> /home/minecraft/minecraft/RestartLog.txt > /dev/null
It is a good idea to keep a log of the restarts. In this line, you make the execution of the ./start.sh dependent on the fact that the cd succeeds. Also, the echo only gets executed after the ./start.sh exists.
So that leaves me with a question: does start.sh keep on running as long as the server runs (in that case: the ps-test is ok, but the && echo makes no sense, or does start.sh exit while leaving the minecraft-server in the background (in that case the ps-grep won't work correctly, but it makes sense to echo the log record only if start.sh exits correctly).
fi
(no remarks for the fi)
If start.sh blocks until the server exists/crashes, you'd be better off to simply restart it in an infinite loop without the involvement of cron. Simply type in a console (or put into another script):
#!/bin/bash
cd /home/minecraft/minecraft/
while sleep 3; do
echo "$(date) server (re)start" >> restart.log
./start.sh # blocks until server crashes
done
But if it doesn't block (i.e. if start.sh starts the server and then returns, but the server keeps running), you would need to implement a different check to verify if the server is actually still running, other than ps|grep start.sh
PS: To kill the infinite loop you have to Ctrl+C twice: Once to stop ./start.sh and once to exit from the immediate sleep.
You can use monit for this task. See docu. It is available on most linux distributions and has a straightforward config. Find some examples in this post
For your app it will look something like
check process minecraftserver
matching "start.sh"
start program = "/home/minecraft/minecraft/start.sh"
stop program = "/home/minecraft/minecraft/stop.sh"
I wrote this answer because sometimes the most efficient solution is already there and you don't have to code anything. Also follow the suggestions of William Pursell and use the init system of your OS (systemd,upstart,system-v,etc.) to host your scripts.
Find more:
Shell Script For Process Monitoring
I have a script that runs a JAVA process that loads data into a database every 10 secs using a loop. This script seems to work perfectly, but after a couple of days I start getting Memory issues. If I stop the script everything frees up, I can start it again and it will run happily for another couple of days.
RUNME=Y
PROPERTIES=someproprties.properties
CHECKFILE=somelockfile.lock
touch $CHECKFILE
while [ "$RUNME" = "Y" ]; do
if [ -f $CHECKFILE ]
then
#Run Process
$DR_HOME/bin/dr -cp $CP_PLUGIN -Xmx64g --engine parallelism=1 --runjson $HOME_DIR/workflows/some_dataflow.dr --overridefile $PROPERTIES 1> /dev/null 2>> $LOG_FILE
#Give Process a little time to finish up before moving on
sleep 10s
else
RUNME=N
fi
done
I had assumed that once the process had run it would make any memory that it had allocated for the process available again, so that the next iteration of the loop could use this. Given that this does not seem to be the case, is there a way I can force the release of memory post the running the process. I appreciate that this may be something that I need to address in the actual JAVA Process rather than in a Shell - but as this is the area I have more control over, I thought I would at least ask.
To check the processes which are running and memory used
sid=$(ps -p $$ -osid=)
while ....
ps --sid $sid -opid,tty,cpu,vsz,etime,command
vsz shows the virtual size used by the process
Then if it's really bash process, it may be environment which is growing, but from the script it can't be that.
For those of you who know what you're talking about I apologise for butchering the way that I'm going to phrase this question. I know nothing about bash whatsoever. With that caveat out of the way, let me get out my cleaver...
I am building a Rails app which has what's called a procfile which sets up any processes that need to be run in different environments
e.g.
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
redis: redis-server
worker: bundle exec sidekiq
proxylocal: bin/proxylocal_local
Each one of these lines specs a process to be run. It also expects a pid to be returned after the process spins up. The syntax is
process_name: process_invokation_script
However the last process, proxylocal, only actually starts a process in development. In production it doesn't do anything.
Unfortunately that causes the Procfile to choke as it needs a process ID returned. So is there some super-simple, zero-overhead process that I can spawn in that case just to keep the procfile happy?
The sleep command does nothing for a specified period of time, with very low overhead. Give it an argument longer than your code will run.
For example
sleep 2147483647
does nothing for 231-1 seconds, just over 68 years. I picked that number because any reasonable implementation of sleep should be able to handle it.
In the unlikely event that that doesn't work (say if you're on an old 16-bit system that can't sleep for more than 216-1 seconds), you can do a sleep in an infinite loop:
sh -c 'while : ; do sleep 30000 ; done'
This assumes that you need the process to run for a very long time; that depends on what your application needs to do with the process ID. If it's required to be unique as long as the application is running, you need something that will continue to run for a long time; if the process terminates, its PID can be re-used by another process.
If that's not a requirement, you can use sleep 0 or true, which will terminate immediately.
If you need to give the application a little time to get the process ID before the process terminates, something like sleep 10 or even sleep 1 might work, though determining just how long it needs to run can be tricky and error-prone.
If Heroku isn't doing anything with proxylocal I'm not sure why you'd even want this in your Procifle. I'm also a bit confused about whether you want to change the Procfile or what bin/proxylocal_local does and how you would even do that.
That being said, if you are able to do anything you like for production your script can just call cat and it will create a pid and then just sit waiting for the next command (which never comes).
For truly minimal overhead, you don't want to run any external commands. When the shell starts a command, it first forks itself, then the child shell execs the external command. If the forked child can run a builtin, you can skip the exec.
Start by creating a read-only fifo somewhere.
mkfifo foo
chmod 400 foo
Then, whenever you need a do-nothing process, just fork a shell which tries to read from the fifo. It's read-only, so no one can write to it, so all reads will block.
read < foo &
I have a program that runs as a daemon, using the C command fork(). It creates a new instance that runs in the background. The main instance exists after that.
What would be the best option to check if the service is running? I'm considering:
Create a file with the process id of the program and check if it's running with a script.
Use ps | grep to find the program in the running proccess list.
Thanks.
I think it will be better to manage your process with supervisord, or other process control system.
Create a cron job that runs every few minutes (or whatever you're comfortable with) and does something like this:
/path/to/is_script_stopped.sh && /path/to/script.sh
Write is_script_stopped.sh using any of the methods that you suggest. If your script is stopped cron will evaluate your script, if not, it won't.
To the question, you gave in the headline:
This simple endless loop will restart yourProgram as soon as it fails:
#!/bin/bash
for ((;;))
do
yourProgram
done
If your program depends on a resource, which might fail, it would be wise to insert a short pause, to avoid, that it will catch all system resources when failing million times per second:
#!/bin/bash
for ((;;))
do
yourProgram
sleep 1
done
To the question from the body of your post:
What would be the best option to check if the service is running?
If your ps has a -C option (like the Linux ps) you would prefer that over a ps ax | grep combination.
ps -C yourProgram