Hi I want Thor to start a server - Jekyll / Python / PHP etc then open the browser
However the starting is a blocking task.
Is there a way to create a child process in Thor; or spawn a new terminal window - couldnt see and google gave me no reasonable answers.
My Code
##
# Project Thor File
#
# #use thor list
##
class IanWarner < Thor
##
# Open Jekyll Server
#
# #use thor ian_warner:openServer
##
desc "openServer", "Start the Jekyll Server"
def openServer
system("clear")
say("\n\t")
say("Start Server\n\t")
system("jekyll --server 4000 --auto")
say("Open Site\n\t")
system("open http://localhost:4000")
say("\n")
end
end
It looks like you are messing things up. Thor is in general a powerful CLI wrapper. CLI itself is in general singlethreaded.
You have two options: either to create different Thor descendents and run them as different threads/processes, forcing open thread/process to wait until jekyll start is running (preferred,) or to hack with system("jekyll --server 4000 --auto &") (note an ampersand at the end.)
The latter will work, but you still are to control the server is started (it may take a significant amount of time.) The second ugly hack to achieve this is to rely on sleep:
say("Start Server\n\t")
system("jekyll --server 4000 --auto &")
say("Wait for Server\n\t")
system("sleep 3")
say("Open Site\n\t")
system("open http://localhost:4000")
Upd: it’s hard to imagine what do you want to yield. If you want to leave your jekyll server running after your script is finished:
desc "openServer", "Start the Jekyll Server"
def openServer
system "clear"
say "\n\t"
say "Starting Server…\n\t"
r, w = IO.pipe
# Jekyll will print it’s running status to STDERR
pid = Process.spawn("jekyll --server 4000 --auto", :err=>w)
w.close
say "Spawned with pid=#{pid}"
rr = ''
while (rr += r.sysread(1024)) do
break if rr.include?('WEBrick::HTTPServer#start')
end
Process.detach(pid) # !!! Leave the jekyll running
say "Open Site\n\t"
system "open http://localhost:4000"
end
If you want to shutdown the jekyll after the page is opened, you are to spawn the call to open as well and Process.waitpid for it.
Related
Using Ruby (tested with versions 2.6.9, 2.7.5, 3.0.3, 3.1.1) and forking processes to handle socket communication there seems to be a huge difference between MacOS OSX and a Debian Linux.
While running on Debian, the forked processes get called in a balanced manner - that mean: if having 10 tcp server forks and running 100 client calls, each fork will get 10 calls. The order of the pid call stack is also always the same even not ordered by pid (caused by load when instantiating the forks).
Doing the same on a MacOS OSX (Catalina) the forked processes will not get called balanced - that mean: "pid A" might get called 23 or whatever times while e.g. "pid G" was never used.
Sample code (originally from: https://relaxdiego.com/2017/02/load-balancing-sockets.html)
#!/usr/bin/env ruby
# server.rb
require 'socket'
# Open a socket
socket = TCPServer.open('0.0.0.0', 9999)
puts "Server started ..."
# For keeping track of children pids
wpids = []
# Forward any relevant signals to the child processes.
[:INT, :QUIT].each do |signal|
Signal.trap(signal) {
wpids.each { |wpid| Process.kill(:KILL, wpid) }
}
end
5.times {
wpids << fork do
loop {
connection = socket.accept
connection.puts "Hello from #{ Process.pid }"
connection.close
}
end
}
Process.waitall
Run some netcat to the server on a second terminal:
for i in {1..20}; do nc -d localhost 9999; done
As said: if running on Linux each forked process will get 4 calls - doing same on MacOS OSX its a random usage per forked process.
Any solution or correction to make it work on MacOS OSX in a balanced manner also?
The problem is that the default socket backlog size is 5 on MacOS and 128 on Linux. You can change the backlog size by passing it as the second argument to TCPServer#listen:
socket.listen(128)
Or you can use the backlog size from the environment variable SOMAXCONN:
socket.listen(ENV.fetch('SOMAXCONN', 128).to_i)
I'm working on a production app that has multiple rails servers behind nginx loadbalancer. We are monitoring sidekiq processes with monit, and it works just fine - when sidekiq proces dies monit starts it right back.
However recently encountered a situation where one of these processes was running and visible to monit, but for some reason not visible to sidekiq. That resulted in many failed jobs and took us some time to notice that we're missing one process in sidekiq Web UI, since monit was telling us everything was fine and all processes were running. Simple restart fixed the problem.
And that bring me to my question: how do you monitor your sidekiq processes? I know i can use something like rollbar to notify me when jobs fail, but i'd like to know if there is a way to monitor process count and preferably send mail when one dies. Any suggestions?
Something that would ping sidekiq/stats and verify response.
My super simple solution to a similar problem looks like this:
# sidekiq_check.rb
namespace :sidekiq_check do
task rerun: :environment do
if Sidekiq::ProcessSet.new.size == 0
exec 'bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e production'
end
end
end
and then using cron/whenever
# schedule.rb
every 5.minutes do
rake 'sidekiq_check:rerun'
end
We ran into this problem where our sidekiq processes had stopped working off jobs overnight and we had no idea. It took us about 30 minutes to integrate http://deadmanssnitch.com by following these instructions.
It's not the prettiest or cheapest option but it gets the job done (integrates nicely with Pagerduty) and has saved our butt twice in the last few months.
On of our complaints with the service is the shortest grace interval we can set is 15 minutes which is too long for us. So we're evaluating similar services like Healthchecks, etc.
My approach is the following:
create a background job that does something
call the job regularly
check that the thing is being done!
so; using a cron script (or something like whenever) every 5 mins, I run :
CheckinJob.perform_later
It's now up to sidekiq (or delayed_job, or whatever active job you're using) to actually run the job.
The job just has to do something which you can check.
I used to get the job to update a record in my Status table (essentially a list of key/value records). Then I'd have a /status page which returns a :500 status code if the record hasn't been updated in the last 6 minutes.
(obviously your timing may vary)
Then I use a monitoring service to monitor the status page! (something like StatusCake)
Nowdays I have a simpler approach; I just get the background job to check in with a cron monitoring service like
IsItWorking
Dead Mans Snitch
Health Checks
The monitoring service which expects your task to check in every X mins. If your task doesn't check in - then the monitoring service will let you know.
Integration is dead simple for all the services. For Is It Working it would be:
IsItWorkingInfo::Checkin.ping(key:"CHECKIN_IDENTIFIER")
full disclosure: I wrote IsItWorking !
I use god gem to monitor my sidekiq processes. God gem makes sure that your process is always running and also can notify the process status on various channels.
ROOT = File.dirname(File.dirname(__FILE__))
God.pid_file_directory = File.join(ROOT, "tmp/pids")
God.watch do |w|
w.env = {'RAILS_ENV' => ENV['RAILS_ENV'] || 'development'}
w.name = 'sidekiq'
w.start = "bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e #{ENV['RAILS_ENV']}"
w.log = "#{ROOT}/log/sidekiq_god.log"
w.behavior(:clean_pid_file)
w.dir = ROOT
w.keepalive
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.interval = 120.seconds
c.above = 100.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
restart.condition(:cpu_usage) do |c|
c.interval = 120.seconds
c.above = 80.percent
c.times = 5
end
end
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 1.hours
end
end
end
I cannot write to a file for the life of me using Sinatra in production.
In my development environment, I can use Logger without a problem and log STDOUT to a file.
It seems like in production, the Logger class is overwritten by the RACK middleware's Logger and it makes things more complicated.
I simply want to write to a file like this:
post '/' do
begin
$log_file = File.open("/home/ec2-user/www/logs/app.log", "w")
...do..stuff...
$log_file.write "INFO -- #{Time.now} --\n #{notification['Message']}"
...do..stuff...
rescue
$log_file.write "ERROR -- #{Time.now} --" + "\njob failed"
ensure
$log_file.close
end
end
The file doesn't get created when I receive a POST request to '/'.
However the file DOES get created when I load the app running pry:
pry -r ./app.rb
I am certain the code inside the POST block is effectively running because new jobs are getting added to the database upon receiving requests..
Any help would be greatly appreciated.
I was finally able to get to the bottom of this.
I changed the nginx user in /etc/nginx/nginx.conf from nginx to ec2-user. (Ideally I would just fix the write permissions for the nginx user but this solution suits me for now.)
Then I ps aux | grep unicorn and saw the timestamp next to the process name: unicorn master -c unicorn.rb -D was 3 days old!!
All this time I was pushing my code the the production server, restarting nginx and never killed and restart the unicorn process.
I removed all the code in my POST block and left only the file creation part
post '/' do
$log_file = File.open("/home/ec2-user/www/logs/app.log", "a")
$log_file.write("test log string")
$log_file.close
end
And the the file was successfully written to upon receiving a POST request.
Due to unicorn_rails complaining about different gem versions we moved to running bundle exec unicorn_rails... in our bluepill files. This change solved that particular problem and things start and stop but when we try sudo bluepill status we now get
unicorn(pix: XXXXXX): unmonitored
Which looks like bluepill is not monitoring the unicorn processes now. It will restart the child processes if I stop them but won't restart the parent process.
I've searched around but can't find much about this issue and was hoping someone could shed some light on it. The bluepill config file is
app_dir = "/opt/local/share/httpd/apps/xyz"
Bluepill.application('xyz', :log_file => "#{app_dir}/current/log/bluepill.log") do |app|
app.process('unicorn') do |process|
process.pid_file = "#{app_dir}/shared/pids/unicorn.pid"
process.working_dir = "#{app_dir}/current"
process.stdout = process.stderr = "#{app_dir}/shared/log/unicorn.err.log"
process.start_command = "bundle exec unicorn_rails -D -c #{app_dir}/current/config/environments/production/unicorn.rb -E production"
process.stop_command = "kill -QUIT {{PID}}"
process.restart_command = "kill -USR2 {{PID}}"
process.start_grace_time = 8.seconds
process.stop_grace_time = 5.seconds
process.restart_grace_time = 13.seconds
process.monitor_children do |child_process|
child_process.stop_command = "kill -QUIT {{PID}}"
child_process.checks :mem_usage, :every => 10.seconds, :below => 200.megabytes, :times => [3,5]
child_process.checks :cpu_usage, :every => 10.seconds, :below => 50, :times => [3,5]
end
end
end
When you run bundle exec, it sets up an environment and forks the unicorn_rails process. Bluepill ends up monitoring the original bundle exec process instead of unicorn which is why you see unmonitored.
I set up my bundler environment directly in bluepill and then execute unicorn_rails directly:
Bluepill.application('xyz') do |app|
app.environment = `env -i BUNDLE_GEMFILE=#{app_dir}/Gemfile bundle exec env`.lines.inject({}) do |env_hash,l|
kv = l.chomp.split('=',2)
env_hash[kv[0] = kv[1]
env_hash
end
app.process('unicorn') do |process|
process.start_command = "unicorn_rails -D -c #{app_dir}/current/config/environments/production/unicorn.rb -E production"
end
end
(Note: I omitted part of the above config file for clarity. Your config file looks good, just try adding the app.environment stuff above and removing bundle exec from your start command.)
This sets up the environment in bluepill by capturing the bundler environment variables using backticks, parsing the returned string into a hash and assigning it to app.environment.
I know this question is old, but I've been facing this problem for weeks. My suspicions were aroused when I realised that 'bluepill quit', followed by reloading the pill while unicorn was running allowed bluepill to consider the process 'up'.
#blt04's answer didn't help. Today I came to a realization. My grace start time of 8 seconds was not enough, because I had preload_app true in my unicorn config... and my app (Rails) takes 12 seconds to load, not 8.
Raising the start time to 30 seconds (wildly in excess of what was needed) solved the problem. Bluepill just says 'starting' for 30 seconds, then goes to 'up' correctly. Unicorn starts and runs as normal.
You'll want your restart time to be longer than it takes for Rails to start too.
I have written a ruby script which opens up dlink admin page in firefox and does a ADSL connection or disconnection.
I could run this script in the terminal without any problem. But if I put it as cron job, it doesn't fire up firefox.
This is the entry I have in crontab
# connect to dataone
55 17 * * * ruby /home/raguanu/Dropbox/nettie.rb >> /tmp/cron_test
I see the following entries in /tmp/cron_test. So it looks like the script indeed ran.
PROFILE:
i486-linux
/usr/bin/firefox -jssh
But I couldn't figure out why I didn't see firefox opening up, for this automation to work. Here is /home/raguanu/Dropbox/nettie.rb
#!/usr/bin/ruby -w
require 'rubygems'
require 'firewatir'
require 'optiflag'
module Options extend OptiFlagSet
character_flag :d do
long_form 'disconnect'
description 'Mention this flag if you want to disconnect dataone'
end
flag :l do
optional
long_form 'admin_link'
default 'http://192.168.1.1'
description 'Dlink web administration link. Defaults to http://192.168.1.1'
end
flag :u do
optional
long_form 'user'
default 'admin'
description 'Dlink administrator user name. Defaults to "admin"'
end
flag :p do
optional
long_form 'password'
default 'admin'
description 'Dlink administrator password. Defaults to "admin"'
end
flag :c do
optional
long_form 'connection_name'
default 'bsnl'
description 'Dataone connection name. Defaults to "bsnl"'
end
extended_help_flag :h do
long_form 'help'
end
and_process!
end
class DlinkAdmin
include FireWatir
def initialize(admin_link = "http://192.168.1.1", user = 'admin', pwd = 'admin')
#admin_link, #user, #pwd = admin_link, user, pwd
end
def connect( connection_name = 'bsnl' )
goto_connection_page connection_name
# disconnect prior to connection
#browser.button(:value, 'Disconnect').click
# connect
#browser.button(:value, 'Connect').click
# done!
#browser.close
end
def disconnect( connection_name = 'bsnl' )
goto_connection_page connection_name
# disconnect
#browser.button(:value, 'Disconnect').click
# done!
#browser.close
end
private
def goto_connection_page( connection_name = 'bsnl')
#browser ||= Firefox.new
#browser.goto(#admin_link)
# login
#browser.text_field(:name, 'uiViewUserName').set(#user)
#browser.text_field(:name, 'uiViewPassword').set(#pwd)
#browser.button(:value,'Log In').click
# setup > dataone
#browser.image(:alt, 'Setup').click
#browser.link(:text, connection_name).click
end
end
admin = DlinkAdmin.new(Options.flags.l, Options.flags.u, Options.flags.p)
unless Options.flags.d?
admin.connect( Options.flags.c )
else
admin.disconnect( Options.flags.c )
end
Any help is appreciated.
You need to have a DISPLAY environment pointing at a valid X-server. This could either involve setting it to the value ":0.0" (without quotes), such that it refers to your local standard DISPLAY.
There's a few things to keep in mind though:
You could run an X virtual frame buffer (xvfb), so that Firefox simply uses that as it's display. This would mean that Firefox would be able to do all its graphical operations, but that it would be independent of your standard graphical environment. You'll have to set the DISPLAY variable appropriately so that it points to the xvfb instance. For instance, if you invoke xvfb as follows:
Xvfb :1 -screen 0 1600x1200x32
Then you'll be able to use this by setting the DISPLAY variable to :1
You're starting a full-blown firefox instance to simply connect or disconnect your modem. You would most likely be able to use "curl" to send the appropriate HTTP requests to the server, such that it performs a connect or disconnect for you. One way to trivially see what you should recreate would be to install a Firefox plugin such as LiveHTTPHeaders and note down the most important HTTP requests as you perform the actions manually.
There's even a ruby binding for curl:
libcurl for Ruby. The resulting script should be much smaller than your current script.
Programs run from cron don't have your interactive environment. Therefore they don't have and DISPLAY variable, and so you can't run any X (graphical) programs, e.g. Firefox.
I would suggest doing the HTTP connections yourself, in ruby, rather than trying to automate Firefox.
the crontab entry is wrong
it is like
#min hour day month dow user command
55 17 * * * ur_user_is_missing ruby /home/raguanu/Dropbox/nettie.rb >> /tmp/cron_test