Sometimes I need to run on specific MIQ worker in foreground.
rake evm:start runs all the workers, but if I need just one, how can I do that?
In case if you are unsure what workers to work with, you might be able to do the following:
run evm server normally: bundle exec rake evm:start and see what worker types were running: bundle exec rake evm:status
kill evm server: bundle exec rake evm:stop
start a single worker in the foreground:
ruby lib/workers/bin/run_single_worker.rb MiqWorkerClassHere
Related
I am running a puma server ruby application on fedora 32. In my server I have certain calls which will spawn new long running processes for various reasons. I came across an issue where my spawned processes were running and listening on the same port as my server. This lead to issues with restarting my server on deploys as the server could not start because of processes listening on the desired port. How could this be possible? From my understanding when I spawn a process it should have completely different memory to the parent process, and share no file descriptors. My spawn command is simply
my_pid = Process.spawn(my_cmd, %i[out err] => log_file)
Ruby version 2.7.0
Edit: something I had overlooked in my deploy process and my original problem description, server restart is not an actual tear down and restart of a new process, but via signalling USR2 to the puma server (as described here)
A quick workaround / solution will be to call fork, close Puma's socket within the forked process and then call exec, which replaces the running process... however, this workaround is limited to Unix systems. On windows you can probably achieve something similar using a more complicated approach.
Sadly, I am not sure how to close Puma's listening socket. Perhaps this will helps, but more likely than not there's some other trick to this.
I believe I have found what is causing this. Seems to be an issue with puma restart process, which I was using. By restarting the server with a USR2 signal, it changes the flags on the open fd for the socket.
[me#home puma_testing]$ cat /proc/511620/fdinfo/5
pos: 0
flags: 02000002
mnt_id: 10
[me#home puma_testing]$ kill -s USR2 511620
[me#home puma_testing]$ cat /proc/511620/fdinfo/5
pos: 0
flags: 02
mnt_id: 10
This was tested on fedora 32 using a very simple puma and sinatra setup like so:
puma.rb
# frozen_string_literal: true
rackup File.join(File.dirname(File.realpath(__FILE__)), './server.ru')
# https://www.rubydoc.info/gems/puma/Puma/DSL#prune_bundler-instance_method
# This allows us to install new gems with just a phased-restart. Otherwise you
# need to take the master process down each time.
prune_bundler
port 11111
environment 'production'
pidfile File.join(File.dirname(File.realpath(__FILE__)), '../', 'server.pid')
tag 'test'
And server.ru like so
require 'sinatra'
class App < Sinatra::Base
get "/" do
"Hello World!"
end
get "/spawn" do
spawn "sleep 500"
end
end
run App
Ran using bundler bundle exec puma -C puma.rb. Note you can use /spawn get request to test spawning a new process before and after restart to see if it is listening on the socket with lsof -itcp:11111
Pardon my ignorance, but is there to increase the number of processes per dyno for Resque workers? And if so, how?
I'm currently using Unicorn to add concurrency to the web dynos, which has been working great so far. I would like to extend this to the Resque workers. I followed Heroku's guide to set up the concurrency.
Update: The solution below works, but is not recommended. For resque concurrency on heroku use the resque-pool gem.
It is possible if you use the COUNT=* option. Your procfile will look something like:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
resque: env TERM_CHILD=1 COUNT=2 RESQUE_TERM_TIMEOUT=6 QUEUE=* bundle exec rake resque:workers
It is impotant to note that the rake task in the Procfile is resque:workers and not resque:work.
Update Explanation
There are major problems with the COUNT=* option and the rake resque:workers invocation in production on heroku. Because of the way resque starts up the multiple workers using threads, all of the SIGTERM, SIGKILL, etc. handling that allows workers to stop the current job, re-enqueue job, and shut down properly (including de-registering) will ever happen. This is because the the signals are handled by the main process not trapped by the threads. This can cause phantom workers to remain in the worker list long after they've been killed. This is probably why there is a comment in the resque code that warns that the resque:workers should only be used in development mode.
I'm trying to execute a ruby daemon framework-less app on Heroku, but it crash after trying to execute the worker:
heroku[worker.1]: State changed from crashed to starting
heroku[worker.1]: Starting process with command `bundle exec rake twitter:start`
heroku[worker.1]: State changed from starting to up
heroku[worker.1]: Process exited with status 0
heroku[worker.1]: State changed from up to crashed
The app basically collects data, through a bunch of APIs, and saves it on a remote MongoHQ instance.
The rake tasks are:
dir_path = File.expand_path('../', __FILE__)
namespace :twitter do
desc 'Start Twitter daemon'
task :start do
exec "#{dir_path}/bin/autana start"
end
...
end
The Procfile is very simple: worker: bundle exec rake twitter:start
The code that executes the daemon is the following:
#!/usr/bin/env ruby
require 'daemons'
require File.expand_path('../../lib/autana', __FILE__)
Daemons.run_proc('autana', multiple: true, no_pidfiles: true) do
client = Autana::Client::Twitter.new
client.collect
end
Any ideas?
Can you run it dynamically?
heroku run rake twitter:start
So, it was a MongoDB error. Data was not properly associated. After entering the console, and changed the document, everything worked as expected.
I'm currently trying to monitor various processes/daemons of in total three Rails/Rack Applications using god. Monitoring works great, the problem is that i'm not able to configure god to autostart all processes after a reboot.
My Setup: I'm running a Linux VPS with Centos & Plesk.
I have a non-root linux user "deployer" which is used to deploy & run the three Rails/Rack Applications. Two applications are running with the passenger apache module, the third application uses a thin Server (that's necessary because the application doesen't work with apache). The two Rails applications, that are using passenger have additional rake tasks that run in the background - these and the thin Server are monitored by god.
The god gem is specified in the Gem File of all three Applications.
In every deploy.rb file i have a method that looks like
namespace :misc do
desc "restart woekers using gog; restart webserver"
task :restart, roles: [:web, :resque] do
run "touch #{current_path}/tmp/restart.txt"
god.all.start
god.all.reload
god.all.terminate
god.all.start
end
end
After a reboot of the server, if i run the cap misc:restart for all three applications manually, all processes are booted up and monitored correctly.
Every try to start god automatically on boot and start all necessary processes failed so far.
I tried many different things, but nothing worked. My approach so far was to create a cron task with #reboot that runs three of the following script:
#!/bin/bash -l
cd /path/to/app/ && bundle exec god -c /path/to/app/config/god/resque.god && bundle exec god load /path/to/app/config/god/resque.god && bundle exec god start resque
This works great for the first application: god and all processes are started.
When the script is executed for the second application (of course with the with the correct paths), god is not able to start the tasks.
I enabled logging in god and the error message (in case of the Rack Application) was "thin: command not found".
When I'm starting the Rack Application first, thin is started correctly and the commands of the other task are not found.
I don't get whats wrong with my configuration. I added the bundle exec command in front of the god calls as you can see above (so the commands should be executed in the environment of their respective application) - nevertheless, it just doesen't work.
I would really appreciate if anyone could help me getting god to start automatically.
If you need further information please don't hesitate to ask!
Thanks in Advance!
Am working on something similar and took this approach:
Use upstart or something similar to launch the god daemon on system boot, for me this is done like so:
/etc/init/god.conf
description "god"
start on runlevel [2]
stop on runlevel [016]
console owner
exec /usr/local/rvm/bin/rvm_god -c /etc/god
respawn
That guy runs god specifying one ruby god configuration file with the -c option:
/etc/god
# Load the configs
God.load "/home/dangerousbeans/kitten_smusher/config/config.god"
God.load "/home/dangerousbeans/irc_nommer/config/config.god"
This ruby dude loads in the individual application god configs and running God.load causes them to boot up.
The individual files look like this I guess as I'm using RVM:
/home/dangerousbeans/irc_nommer/config/config.god
God.watch do |w|
w.dir = "/home/dangerousbeans/irc_nommer"
w.name = "IRCnommer"
# scary rvm magic begins
gemsets_path = [
"/home/dangerousbeans/.rvm/gems/ruby-1.9.3-p125#irc_nommer/bin",
"/home/dangerousbeans/.rvm/rubies/ruby-1.9.3-p125/bin",
"/home/dangerousbeans/.rvm/bin",
ENV['PATH'] # inherit this
].join(':')
w.env = {
"PATH" => gemsets_path,
"GEM_PATH" => "/home/dangerousbeans/.rvm/gems/ruby-1.9.3-p125#irc_nommer"
}
# scary rvm magic ends
w.log = "/tmp/ircnommer.log"
w.start = "ruby /home/dangerousbeans/irc_nommer/irc_nommer.rb"
w.keepalive
end
The key point is the environments is different between manual and automatic while god execute the [start] command.
So you can add command env to the command. like:
God.watch do |w|
w.start = "cd #{your_app_directory}; env >> log/god.log; your-real-command >> log/god.log 2>&1"
end
There'll be some differences as you type env in the same directory.
Check the difference and add required/correct paragraph to god's env.
Today I encounter an issue, I deployed 2 rails apps in 1 server, both uses god. The App#2 can't startup the command correctly. After do above test I found the cause: God hold an environment variable [BUNDLE_GEMFILE] that points to App#1. So I add a simple line then error gone away:
God.watch do |w|
w.env = {
"BUNDLE_GEMFILE" => "#{$rails_root}/Gemfile"
}
end
I have a few Resque jobs running, each started in a separate terminal window like so:
QUEUE=queue_1 rake environment resque:work
QUEUE=queue_2 rake environment resque:work
Queue 1 started first, then queue 2. The problem is, no matter what QUEUE options I send to new workers, they just keep working on queue 1 -- even if I shut both down. Might this a configuration problem? I haven't seen this issue mentioned anywhere.
Are you explicitly setting the ENV['QUEUE'] environment variable in the "environment" or "resque:setup" tasks defined in the Rakefile?