I have a ruby 2.0 sinatra 'faceless' app that serves up json by calling an external service. It works fine.
The main app is run on port 80 in a ubuntu machine.
I also start an instance using 'foreman start' - so it runs on port 5000 on the same ubuntu virtual machine.
On the port 80 instance, the process 'foreman master' soaks up CPU time, while with the same load, the one on port 5000 uses essentially 0 CPU.
$ ps -a l
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 0 1615 1614 20 0 26140 17236 wait Sl+ tty1 0:28 foreman: master
0 1000 1899 1659 20 0 25036 16612 wait Sl+ pts/1 0:00 foreman: master
The apps were started at the same time and both had the same load (very light for 20 mins).
The only difference I can see is that the problem one is started on port 80 using a sudo command, and the other one is just started as a user process.
Is there a difference in how foreman needs to output log entries in a tty terminal vs a pts/1 terminal?
Note that with 40 people banging away on the app, the foreman master process is using 90% cpu while all the other ruby processes that are supposed to be doing the work are at 1% (9 unicorns).
I think its something to do with terminal output handled differently but I'm not sure.
Thanks for any help.
Is there a way to tell foreman or ruby to not write log stuff out at all?
EDIT
I now think that it is related to terminal logging, since i turned 95% of it off for the deployment app, and loads are better, but still higher than the normal non rvmsudo command.
Related
I'm trying to use Elasticsearch in Docker for local dev. While I can find containers that work, when docker stop is sent, the containers hang for the default 10s, then docker forcibly kills the container. My assumption here is that ES is either not on PID 1 or other services prevent it from shutting down immediately.
I'm curious if anyone can expand on this, or explain why this is happening more accurately. I'm running numerous tests and 10s+ to shutdown is just annoying when other containers shutdown after 1-2s.
If you don't want to wait the 10 seconds, you can run a docker kill instead of a docker stop. You can also adjust the timeout on docker stop with the -t option, e.g. docker stop -t 2 $container_id to only wait 2 seconds instead of the default 10.
As for why it's ignoring the sigkill, that may depend on what image you are running (there's more than one for elasticsearch). However, if pid 1 is a shell like /bin/sh or /bin/bash, it will not pass signals through. If pid 1 is the elasticsearch process, it may ignore the signal, or 10 seconds may not be long enough for it to fully cleanup and shutdown.
I need to call commands sporadically in my ruby app, most commands will end quite quickly, but some will run for longer (like playing a song) and I need to be able to kill that process if wanted.
The commands are called in a sub thread as not to lock the rest of the app.
I also need to store info on running commands (so I know what to kill if needed), but I also need to send out info when the command ended.
The problems:
wait or waitpid doesn't work (they never return)
I tried to use trap, which detects when the command ended, but it only works for the latest started command.
Ruby leaves the commands as zombie process (until killing the app) and trying to use the kill(both Process.kill and calling the command directly) doesn't remove them. (Perhaps the reason to why wait doesn't work??)
# Play sound (player can be either 'mplayer' or 'aplay' )
pid = spawn player, sound_dir + file
Eddie::common[:commands][:sound_pids][pid] = file
Signal.trap("CLD") {
# This runs, but doesn't clean up zombies
Process.kill "KILL", pid
p = system 'kill ' + pid.to_s
Eddie::common[:commands][:sound_pids].delete pid
}
If I run this twice (once while the first command is running) it will look like this after both commands ended:
Eddie::common[:commands][:sound_pids] => {3018=>"firstfile.wav"}
and this is the result of ps
chris 3018 0.0 0.0 0 0 pts/5 Z+ 23:50 0:00 [aplay] <defunct>
chris 3486 0.2 0.0 0 0 pts/5 Z+ 23:51 0:00 [aplay] <defunct>
Added note: Using system to call the command doesn't leave a zombie process, but in turn it's not killable..
I created monit app that must restart golang site on crash
$ cd /etc/monit/conf.d
$ vim checkSite
It starting program with nohup and saving its pid to file:
check process site with pidfile /root/go/path/to/goSite/run.pid
start program = "/bin/bash -c 'cd /root/go/path/to/goSitePath; nohup ./goSite > /dev/null 2>&1 & echo $! > run.pid'" with timeout 5 seconds
stop program = "/bin/kill -9 `cat /root/go/path/to/goSitePath/run.pid`"
It starts ok.
Process 'site'
status Running
monitoring status Monitored
pid 29723
parent pid 1
uptime 2m
children 0
memory kilobytes 8592
memory kilobytes total 8592
memory percent 0.4%
memory percent total 0.4%
cpu percent 0.0%
cpu percent total 0.0%
data collected Thu, 05 Mar 2015 07:20:32
Then to test how it will restart on crash I killed manually golang site.
Here I have two issues:
Site is restarted rather slow: it takes 1 minute although in configuration I set with timeout 5 seconds
Status of site in monit becomes Does not exist even after site in fact restarts. I guess this occurs because after killing and restarting site's pid is changing randomly, but how to overcome this I don't know.
status after restart:
Process 'site'
status Does not exist
monitoring status Monitored
data collected Thu, 05 Mar 2015 08:04:44
How to reduce the time of restarting and how to repair site's monit status?
monit log:
[Mar 5 08:04:44] error : 'site' process is not running
[Mar 5 08:04:44] info : 'site' trying to restart
[Mar 5 08:04:44] info : 'site' start: /bin/bash
[Mar 5 08:06:44] info : 'site' process is running with pid 31479
Update
My golang site is rather simple:
package main
import (
"fmt"
"github.com/go-martini/martini"
)
func main() {
m := martini.Classic()
m.Get("/", func() {
fmt.Println("main page")
})
m.Run()
}
Update 2
I tried to increase speed of monit reload my golang site by removing pid file itself. Say I made kill 29723 && rm run.pid and turned timer on to count time for site been accessible again. It took 85 seconds. So removing pid file did not help monit to increase speed of reloading site.
monit doesn't have any subscription mechanism to inmediatelly discover if a process has died.
In daemon mode, as documented, monit works by periodically polling the status of all the configured rules, its poll-cycle is configured when daemon starts and defaults in some Linux distributions to 2 minutes, what means that in this case, monit can need till 2 minutes to take any action.
Check this configuration in your monitrc, it's configured with the set daemon directive, for example, if you want to check the status every 5 seconds, then you should set:
set daemon 5
On every cycle it updates its status, and executes actions if needed depending on this. So if it detects that the process doesn't exist, it will report Does not exist till the next poll cycle, even if it already takes the decission to restart it.
The timeout in the start daemon directive doesn't have anything to do with this poll-cycle, this is the time monit will give to the service to start. If the service doesn't start in this time monit will report it.
If monit doesn't meet your requirements, you can also try supervisord, that is always aware of the state of the executed programs.
I have a node.js application that uses some socket ports. Sometimes, when I exit the application with Ctrl + C, some kind of node process is left running, still allocating the socket ports. Thus, I cannot restart my application, since it fails to open those ports. When I look the situation with ps, I get the following response:
$ ps
PID TTY TIME CMD
40454 ttys000 0:00.11 -bash
41643 ttys001 0:00.00 (node)
41741 ttys001 0:00.00 (node)
Trying kill -9 41643 doesn't kill the process. Is it a some kind of unkillable zombie? How can I get rid of those (node)-things blocking my tcp ports?
I'm not a MAC user, but here is what I use to kill all the available node processes (under linux):
sudo killall -9 node
On macOS, it's simply:
sudo killall -9 node
For a lot of the times, sudo is overkill, but in your case, it looks like you might want to try sudo.
I am trying to do something really basic;
Plot an array of integers as time-series data using JFreeChart. Previously the code was working perfectly fine. That is the point that drives me crazy.
However, now it fails to terminate after doing everything it is supposed to do. Active threads are as follows;
Thread[AWT-Shutdown,5,main]
Thread[AWT-EventQueue-0,6,main]
Thread[main,5,main]
I am using a Macbook with Mac OS 10.6.8 on it, and got a set of recent software updates.
Does anybody have any clue about where to start, and what to look for?
You can get more infomation from the terminal:
Launch your program in the background
$ java -jar dist/program.jar &
Get its process id
$ ps
PID TTY TIME CMD
714 ttys000 0:00.01 -bash
727 ttys000 0:01.52 /usr/bin/java -jar dist/program.jar
Obtain a thread dump
$ kill -QUIT 727
Look for anything not in State: WAITING or State: RUNNABLE
Your IDE's profiler may offer a more friendly view of the same information. See also JLS ยง12.8 Program Exit. An sscce may help, too.