I'm currently trying to monitor various processes/daemons of in total three Rails/Rack Applications using god. Monitoring works great, the problem is that i'm not able to configure god to autostart all processes after a reboot.
My Setup: I'm running a Linux VPS with Centos & Plesk.
I have a non-root linux user "deployer" which is used to deploy & run the three Rails/Rack Applications. Two applications are running with the passenger apache module, the third application uses a thin Server (that's necessary because the application doesen't work with apache). The two Rails applications, that are using passenger have additional rake tasks that run in the background - these and the thin Server are monitored by god.
The god gem is specified in the Gem File of all three Applications.
In every deploy.rb file i have a method that looks like
namespace :misc do
desc "restart woekers using gog; restart webserver"
task :restart, roles: [:web, :resque] do
run "touch #{current_path}/tmp/restart.txt"
god.all.start
god.all.reload
god.all.terminate
god.all.start
end
end
After a reboot of the server, if i run the cap misc:restart for all three applications manually, all processes are booted up and monitored correctly.
Every try to start god automatically on boot and start all necessary processes failed so far.
I tried many different things, but nothing worked. My approach so far was to create a cron task with #reboot that runs three of the following script:
#!/bin/bash -l
cd /path/to/app/ && bundle exec god -c /path/to/app/config/god/resque.god && bundle exec god load /path/to/app/config/god/resque.god && bundle exec god start resque
This works great for the first application: god and all processes are started.
When the script is executed for the second application (of course with the with the correct paths), god is not able to start the tasks.
I enabled logging in god and the error message (in case of the Rack Application) was "thin: command not found".
When I'm starting the Rack Application first, thin is started correctly and the commands of the other task are not found.
I don't get whats wrong with my configuration. I added the bundle exec command in front of the god calls as you can see above (so the commands should be executed in the environment of their respective application) - nevertheless, it just doesen't work.
I would really appreciate if anyone could help me getting god to start automatically.
If you need further information please don't hesitate to ask!
Thanks in Advance!
Am working on something similar and took this approach:
Use upstart or something similar to launch the god daemon on system boot, for me this is done like so:
/etc/init/god.conf
description "god"
start on runlevel [2]
stop on runlevel [016]
console owner
exec /usr/local/rvm/bin/rvm_god -c /etc/god
respawn
That guy runs god specifying one ruby god configuration file with the -c option:
/etc/god
# Load the configs
God.load "/home/dangerousbeans/kitten_smusher/config/config.god"
God.load "/home/dangerousbeans/irc_nommer/config/config.god"
This ruby dude loads in the individual application god configs and running God.load causes them to boot up.
The individual files look like this I guess as I'm using RVM:
/home/dangerousbeans/irc_nommer/config/config.god
God.watch do |w|
w.dir = "/home/dangerousbeans/irc_nommer"
w.name = "IRCnommer"
# scary rvm magic begins
gemsets_path = [
"/home/dangerousbeans/.rvm/gems/ruby-1.9.3-p125#irc_nommer/bin",
"/home/dangerousbeans/.rvm/rubies/ruby-1.9.3-p125/bin",
"/home/dangerousbeans/.rvm/bin",
ENV['PATH'] # inherit this
].join(':')
w.env = {
"PATH" => gemsets_path,
"GEM_PATH" => "/home/dangerousbeans/.rvm/gems/ruby-1.9.3-p125#irc_nommer"
}
# scary rvm magic ends
w.log = "/tmp/ircnommer.log"
w.start = "ruby /home/dangerousbeans/irc_nommer/irc_nommer.rb"
w.keepalive
end
The key point is the environments is different between manual and automatic while god execute the [start] command.
So you can add command env to the command. like:
God.watch do |w|
w.start = "cd #{your_app_directory}; env >> log/god.log; your-real-command >> log/god.log 2>&1"
end
There'll be some differences as you type env in the same directory.
Check the difference and add required/correct paragraph to god's env.
Today I encounter an issue, I deployed 2 rails apps in 1 server, both uses god. The App#2 can't startup the command correctly. After do above test I found the cause: God hold an environment variable [BUNDLE_GEMFILE] that points to App#1. So I add a simple line then error gone away:
God.watch do |w|
w.env = {
"BUNDLE_GEMFILE" => "#{$rails_root}/Gemfile"
}
end
Related
I am running a puma server ruby application on fedora 32. In my server I have certain calls which will spawn new long running processes for various reasons. I came across an issue where my spawned processes were running and listening on the same port as my server. This lead to issues with restarting my server on deploys as the server could not start because of processes listening on the desired port. How could this be possible? From my understanding when I spawn a process it should have completely different memory to the parent process, and share no file descriptors. My spawn command is simply
my_pid = Process.spawn(my_cmd, %i[out err] => log_file)
Ruby version 2.7.0
Edit: something I had overlooked in my deploy process and my original problem description, server restart is not an actual tear down and restart of a new process, but via signalling USR2 to the puma server (as described here)
A quick workaround / solution will be to call fork, close Puma's socket within the forked process and then call exec, which replaces the running process... however, this workaround is limited to Unix systems. On windows you can probably achieve something similar using a more complicated approach.
Sadly, I am not sure how to close Puma's listening socket. Perhaps this will helps, but more likely than not there's some other trick to this.
I believe I have found what is causing this. Seems to be an issue with puma restart process, which I was using. By restarting the server with a USR2 signal, it changes the flags on the open fd for the socket.
[me#home puma_testing]$ cat /proc/511620/fdinfo/5
pos: 0
flags: 02000002
mnt_id: 10
[me#home puma_testing]$ kill -s USR2 511620
[me#home puma_testing]$ cat /proc/511620/fdinfo/5
pos: 0
flags: 02
mnt_id: 10
This was tested on fedora 32 using a very simple puma and sinatra setup like so:
puma.rb
# frozen_string_literal: true
rackup File.join(File.dirname(File.realpath(__FILE__)), './server.ru')
# https://www.rubydoc.info/gems/puma/Puma/DSL#prune_bundler-instance_method
# This allows us to install new gems with just a phased-restart. Otherwise you
# need to take the master process down each time.
prune_bundler
port 11111
environment 'production'
pidfile File.join(File.dirname(File.realpath(__FILE__)), '../', 'server.pid')
tag 'test'
And server.ru like so
require 'sinatra'
class App < Sinatra::Base
get "/" do
"Hello World!"
end
get "/spawn" do
spawn "sleep 500"
end
end
run App
Ran using bundler bundle exec puma -C puma.rb. Note you can use /spawn get request to test spawning a new process before and after restart to see if it is listening on the socket with lsof -itcp:11111
I have a Rails 5 application which I was planning to deploy on Linux, but because we needed some access very specific Windows-only software, I need to deploy it on Windows Server 2012 R2. My software stack (or mix) was supposed to be Nginx/Puma/Rails/PostgreSQL/Redis. Everything installs for me on Windows except Puma, and the Rails documentation says that I need Puma for ActionCable.
How do I get Puma to run on Windows? I have seen and tried snippets of things to try, and I have also seen and tried snippets on what not to do, such as running in daemon mode because fork() is not supported. Does anybody have a repeatable set of instructions on how to get Puma to work on Windows with a Rails application?
Or, if Puma a non-starter for Windows, is there a repeatable alternative for deploying a Rails 5 application with ActionCable to a Windows Server host (e.g. Windows 2012 R2)?
According to the readme file from the github page, following things to keep in mind:
daemon mode is not supported. so comment out/remove the following, if there is such line.
daemonize false
Workers do not work in Windows since it does not support processes. We want the workers to be "0". So comment out following lines:
workers 2 # The default is "0"
preload_app!
server sockets are not seamless on restart, they must be closed and reopened. These platforms have no way to pass descriptors into a new process that is exposed to ruby.
Do not use unix socket, instead bind the server to "tcp://". So comment out any line that looks like following:
bind 'unix:///var/run/puma.sock'
bind 'unix:///var/run/puma.sock?umask=0111'
Instead use following:
bind "tcp://127.0.0.1:4001"
# You don't have to if you don't need to specify a port
# since the default is "tcp://0.0.0.0:9292"
If you see any http parse error (malformed http request) after starting rails server, try this answer. If it doesn't work, then comment out this line from config/environments/production.rb or config/environments/production.rb (depending on which environment you want to run Puma)
config.force_ssl = true
Here is what the puma.rb file might look like:
worker 0 # Not necessary. The default is "0"
threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i
threads threads_count, threads_count
bind "tcp://127.0.0.1:4001" # Not necessary. Default is "tcp://0.0.0.0:9292"
environment ENV.fetch("RAILS_ENV") { "development" }
plugin :tmp_restart
Finally run bundle exec puma -C config\puma.rb and it should work.
I searched the internet for a "Hello, World" type program for Webrick in Ruby, but could not find anything that worked. I found this guide on SO but for the life of me could not get it to work.
Consulting the Ruby Documentation for Webrick led me to some code snippets that got me going in the right direction. There were no easy-to-follow tutorials so I wanted to add my answer on SO.
I was using Ubuntu 14.04 without Apache or Nginx and wanted my server for a virtual host. Webrick by default does not respond to requests concurrently but for me this was a plus in addition to its small footprint. I was hoping to get it working without the Rails framework for an even lighter footprint.
To get started, I installed Ruby with the Ubuntu package manager. If you are using CentOS or another Linux distribution, you can adapt this step to your particular package manager. Also make sure port 80 is open on your web server. It's possible to get SSL with Webrick but I chose not to at this point.
sudo apt-get install ruby
Here is the script which I named myapp.rb that I am using. I placed it /var/www/myapp. Ideally, I think it should not be in document root. You should also create a special user and group just to run the script to improve security (I have not outlined those steps here)
require 'webrick'
server = WEBrick::HTTPServer.new(:Port => 80,
:SSLEnable => false,
:DocumentRoot => '/var/www/myapp',
:ServerAlias => 'myapp.example.com')
server.mount_proc '/' do |req, res|
res.body = 'Hello, world!'
end
trap 'INT' do server.shutdown end
server.start
The require statement above tells Ruby to include the Webrick classes when running the program. The second line of the script creates an instance of Webrick with the following options:
Use Port 80
Disable SSL
Set the document root to /var/www/myapp
Set the Server Alias to myapp.example.com
Of course, you'll have to configure your particular domains DNS'. The server.mount_proc is telling Ruby to serve the response, "Hello, world" at document root. I think you can specify a subdirectory there if you life. The Ruby Webrick documentation above has information on that.
The line that begins with trap means that the web server can be stopped with Ctrl-C. To start the script I typed the following at the SSH command line:
ruby myapp.rb
I am using puppet to read a fact from facter, and based on that I apply a different configuration to my modules.
Problem:
the puppet agent isn't seeing this fact. Running puppet agent --test interactively works as expected. Even running it non-interactively from a script seems to work fine. Only the agent itself is screwing up.
Process:
I am deploying an Ubuntu-based app stack on EC2. Using userdata (#cloud-config), I set an environment variable in /etc/environment:
export FACTER_tl_role=development
then immediately in #cloud-config, i source /etc/environment.
only THEN i apt-get install puppet (i moved away from using package: puppet to eliminate ambiguity in the sequence of #cloud-config steps)
Once the instance boots, I confirm that the fact is available: running facter tl_role returns "development". I then check /var/log/syslog, and apparently the puppet agent is not seeing this fact - I know this because it's unable to compile the catalog, and there's nothing (blank) where I'm supposed to be seeing the value of the variable set depending on this fact.
However, running puppet agent --test interactively compiles and runs the catalog just fine.
even running this from the #cloud-config script (immediately after installing puppet) also works just fine.
How do I make this fact available to the puppet agent? Restarting the agent service makes no difference, it remains unaware of the custom fact. Rebooting the instance also makes no difference.
here's some code:
EC2 userdata:
#cloud-config
puppet:
conf:
agent:
server: "puppet.foo.bar"
certname: "%i.%f"
report: "true"
runcmd:
- sleep 20
- echo 'export FACTER_tl_role=development' >> /etc/environment
- . /etc/environment
- apt-get install puppet
- puppet agent --test
Main puppet manifest:
# /etc/puppet/manifests/site.pp
node default {
case $tl_role {
'development': { $sitedomain = "dev.foo.bar"}
'production': { $sitedomain = "new.foo.bar"}
}
class {"code" : sitedomain => $sitedomain}
class {"apache::site" : sitedomain => $sitedomain}
class {"nodejs::grunt-daemon" : sitedomain => $sitedomain}
And then I see failures where $sitedomain is supposed to be, so $tl_role appears to be not set.
Any ideas? This is exploding my brain....
Another easy option would be to drop a fact into an external fact.
Dropping a file into /etc/facter/facts.d/* is fairly easy, and you can use a text file, yaml json or an executable to do it.
http://docs.puppetlabs.com/guides/custom_facts.html#external-facts
*that's on Open source puppet, on unix-y machines. See the link for the full docs.
Thank you, #christopher. This may be a good solution, I will test it and possibly move to it from my current horrible hack.
The answer I got in the Puppet Users Google Group was that I should not assume that the Puppet agent process will have an environment of a login shell, and that Facter will also have this environment when it is run by the Puppet agent.
Here is the way I solved it (admittedly, by brute force):
runcmd:
- echo 'export FACTER_tl_role=development' >> /etc/environment
- . /etc/environment
- apt-get install puppet
- service puppet stop
- sed -i '/init-functions/a\. \/etc\/environment' /etc/init.d/puppet
- puppet agent --test
- service puppet start
As you can see, after installing Puppet, I stop the agent, and add a line to /etc/init.d/puppet to source /etc/environment. Then I start the agent. NOT ideal... but it works!
I don't think . /etc/environment is going to work properly the way cloud-init executes runcmd. Two possible solutions:
Export the variable with the puppet agent command:
export FACTER_tl_role=development && puppet agent --test
If that doesn't work:
Just drop the commands into a user-data script and wire them together as a "multipart input" (described in the cloud-init docs).
The second solution executes the commands as a proper shell script, and would likely fix the problem. If the first works, though, it's easier to do with what you have.
Using this example procfile:
web: node app.js
I'm getting errors when running the command
foreman check
The error I'm getting is:
ERROR: no processes defined
Not sure if this matters, but I'm running this on Windows 8. The application runs successively on my local machine running just:
node app.js
Unfortunately, Foreman does not run on Windows. ddollar started up an alternate project, foreman-windows, to try to add Windows support, but I don't know if it ever fully got off the ground.
If you'd like a helper task to start up your node environment locally, writing your own cake task is good, albeit manual, alternative:
{spawn, exec} = require 'child_process'
task 'start', 'Spin up dev environment', ->
exec 'node app.js'
Then you'd execute the task in the command line:
cake start
This gives you the added benefit of creating multiple tasks for various actions, and fine-tuning the tasks to fit the needs of your project.
Note that Windows likes to switch things up when it comes to certain commands. For instance, on *nix-based machines you could open your site in a browser with:
exec "open http://localhost:3000/"
But on Windows, it is start instead:
exec "start http://localhost:3000/"
Another important difference is handling environment variables. On *nix, you can simply prepend them to your command:
exec "NODE_ENV=staging node server.js"
But if you try to use that format with exec on Windows, it'll fail. Instead, spawn a new bash:
terminal = spawn 'bash'
terminal.stdin.write "NODE_ENV=staging node server.js"
terminal.stdin.end()
And you can listen for events from the bash too:
terminal.stdout.on 'data', (data) ->
console.log "#{data}"
terminal.stderr.on 'data', (data) ->
console.log "#{data}"
If you are writing tools that have to support multiple platforms, you can do a OS platform check relatively easily to make accommodations:
os = require 'os'
if os.platform() is 'win32'
<do Windows stuff>