Running resque without Rakefile - ruby

I have built my own job server, which is essentially a private gem, built as a wrapper around resque.
(I am not running this in a Rails environment)
Everywhere I look, it seems like the documented/recommended way to start the workers, is with something like this:
$ QUEUE=* rake resque:work
Which means that it must be executed in a folder where the Rakefile exists.
I am looking for a way to start it without a Rakefile.
What I have learned so far:
I have looked through the issues, maybe someone asked a similar question.
I have looked through the wiki, and specifically the FAQ.
I know I can probably create my own "bin" to run it without rake, by analyzing the tasks file.
I saw that resque installs a resque binary, but it only seems to provide limited functionality, like removing and listing a worker, but not starting.
My current workaround is that my gem's binary is doing chdir to the gem's folder before running (and this folder has a Rakefile), like the code below.
def start_worker
ENV['QUEUE'] = '*'
Dir.chdir gemdir do
exec "rake resque:work"
end
end
def gemdir
File.expand_path "../../", __dir__
end
Appreciate any nudge in the right direction.

The current solution I have worked up for this:
def start_worker
interval = 5
queue = '*'
ENV['QUEUE'] = queue
worker = Resque::Worker.new
Resque.logger = Logger.new STDOUT
Resque.logger.level = Logger::INFO
## this is not yet implemented in 1.26.0, keeping here as a reminder
# worker.prepare
worker.log "Starting worker"
worker.work interval
end
Which is an adaptation of the code from the rake task
For reference, I also opened a github issue, in the off chance that someone else also needs such functionality.

I having created a script to create demon worker processes using following worker starting API.
def start_worker(id)
ENV['QUEUE'] = #queues || "*"
ENV['PIDFILE'] = pid_file(id)
ENV['JOBS_PER_FORK'] = #jobs_per_fork || "1000"
ENV['BACKGROUND'] = 'true'
ENV['TERM_CHILD'] = 'true'
#debug ? ENV['VVERBOSE'] = 'true' : ENV['VERBOSE'] = 'true'
begin
worker = Resque::Worker.new
rescue Resque::NoQueueError
Resque.logger.error "No queue is set for worker_id = #{id}"
end
worker.prepare
worker.log "Starting worker #{self}"
worker.work(5) # interval, will block
end

Related

Running ruby scripts in parallel

Let's say I've got two ruby scripts - a.rb and b.rb. Both are web-scrapers used for different pages. They can work for many, many hours and I would like to run them simultaneously. In order to do that I've tried to run them by third script using 'promise' gem with the following code:
def method_1
require 'path to my file\a'
end
def method_2
require 'path to my file\b'
end
require 'future'
x=future{method_1}
y=future{method_2}
x+y
However this solution throws an error(below) and only one script is executed.
An operation was attempted on something that is not a socket.
(Errno::ENOTSOCK)
I also tried playing with Thread class:
def method_one
require 'path to my file\a'
end
def method_two
require 'path to my file\b'
end
x = Thread.new{method_one}
y = Thread.new{method_two}
x.join
y.join
And it gives me the same error as for 'promise' gem.
I've also run those scripts in separate shells- then they work at the same time, but the performance is much worse (aprox. about 50% slower).
Is it any way to run them at the same time and keep high performance?
You can use concurrent-ruby for this, here is how you can run both your scripts in parallel:
require 'concurrent'
# Create future for running script a
future1 = Concurrent::Promises.future do
require 'path to file\a'
:result
end
# Create future for running script b
future2 = Concurrent::Promises.future do
require 'path to file\b'
:result
end
# Combine both futures to run them in parallel
future = Concurrent::Promises.zip(future1, future1)
# Wait until both scripts are completed
future.value!

Capybara Around Hook to test several envinroments

I'm writing some tests for a webpage that I'd like to run in several environments. The idea is that the test will run in one, then repeat in the next. The two environments are preview and uat.
I've written an Around hook to set the environment variables. Below:
Around do |scenario, block|
def test_envs
chosen_env = ENV['test_env'] || 'preview'
chosen_env.split(',').map(&:strip)
end
test_envs.each do |test_env|
$base_url = "https://#{test_env}.webpage.com"
end
block.call
end
I have then written a method to execute the navigation step:
def navigate_to(path)
visit $base_url + path
end
My Scenario step_definition is:
navigate_to '/login'
The tests will work in either environment, Preview by default or UAT if I set test_env=uat
However, I was aiming to set test_env=preview,uat and have them run consecutively in both environments.
Is there something obvious that I've missed here?
Thanks
If I'm understanding you correctly, it's the 'parallel' aspect that you're asking about.
Rspec can be used with parallel tests (the parallel_tests gem) but I wouldn't be so sure that calling something like 3.times { blk.call } in an around hook will run each block in parallel.
An alternative may be do so some metaprogramming with your example definitions, i.e.
test_envs.each do |env_name|
it "does something in #{env_name}" do
# do something with the specific environment
end
end
Now, I haven't actually used this gem and I don't know for sure it would work. I think the simplest solution may be to just write a wrapper script to call the tests
# run_tests.rb
environments = ENV["TEST_ENV"]&.split(",") || []\
filename = ENV["filename"]
environments.each do |env_name|
Thread.new do
system <<-SH
env TEST_ENV=#{env_name} bundle exec rspec #{filename}
SH
end
end
Running it like env TEST_ENV=foo,bar ruby run_tests.rb would call the following commands in their own threads:
env TEST_ENV=foo bundle exec rspec
env TEST_ENV=bar bundle exec rspec
I like this approach because it means you don't have to touch your existing test code.

when sleep used on a spawned process it kills the process

I am primarily a java guy tinkering with Process.spawn and have hit a bit of a wall. Basically I tossed together a few rspec classes that do puts to tinker with it. This worked pretty well. Then I tossed a sleep in and the whole process just dies when a sleep is reached. I am not sure why, I thought that spawning gets around the shared resource issues of threads. Below is what I was using to make the processes
test_files = Dir["../spec/*.rb"]
test_files.each do |file|
pid = Process.spawn("rspec #{file}")
p pid
end
Process.wait
There is very little going on with the tests in the spec, below is a sample one.
it 'should another thing' do
sleep(0.3)
p "another1"
end
Could someone be so kind as to explain where I am going wrong?
It worked for me, I'm assuming you think it's not working because you get something like this:
script.rb:6:in `wait': No child processes (Errno::ECHILD)
from script.rb:6:in `<main>'
And I'm assuming you get that because you have this script in maybe a lib directory or something, but you're running it from the root of your app, so when it does Dir["../spec/*.rb"] you're expecting it to go up to the root of your app, and then down into the spec directory, but it is based on your pwd, so it goes to the parent directory of the app and looks for a spec directory, which doesn't exist.
Some things I'd advocate in this scenario, check that it finds the files before trying to spawn them. Use absolute filepaths, unless you intentionally want it to be based on where you run the program from. If this is all true, try writing it like this:
spec_glob = File.expand_path('../spec/*.rb', __dir__)
test_files = Dir[spec_glob]
p test_files
test_files.each do |file|
pid = Process.spawn("rspec #{file}")
p pid
end
Process.wait

Resque tasks always fail - uninitialized job constants?

I've tried using Resque before and was met with unmitigated failure. I'm revisiting it again with the same results...
resque.rake:
require "resque/tasks"
task "resque:setup" => :environment
test.rb:
require 'resque'
class FileWorker
#queue = :save_to_file
def self.perform(str)
File.open('./' + Time.now.to_s + '.txt', 'w+') do |f|
f << "test 123"
end
end
end
Resque.enqueue(FileWorker, "12345567".split('').shuffle.join)
Gemfile:
gem 'resque'
gem 'rake'
It seems like running test.rb on its own successfully queues the job:
However, running rake resque:work QUEUE='*' in the same folder results in a warning,
WARNING: This way of doing signal handling is now deprecated. Please
see http://hone.heroku.com/resque/2012/08/21/resque-signals.html for
more info.
As well as the task being added to "failed" queue with the following reason: "exception":"NameError","error":"uninitialized constant FileWorker"
How do I get this to work? Seems like something quite obvious but there's tons of tutorials about Resque spanning many years - some painfully out of date and none explaining how to run workers so they don't fail.
Thanks in advance.
When you enqueue a task with Resque, what is stored on Redis is just the name of the job class (as a string) along with the arguments (again as strings) in a JSON object.
When a worker then tries to perform the task, it needs to be able to create an instance of the job class. It does this by using const_get and const_missing. This is where the error you are seeing occurs, since the worker does not have the definition of FileWorker available to it.
The error is the same as if you tried to get an unknown constant in irb:
> Object.const_missing "FileWorker"
NameError: uninitialized constant FileWorker
The solution is to make sure the definition of FileWorker is available to your workers. The simplest way to do this would be to just require test.rb from your Rakefile (or resque.rake). In your code this would involve adding another task to the queue, so you might want to move the FileWorker code to its own file where it can be required by both the rake file and the code enqueuing jobs.
test.rb:
require 'resque'
require './file_worker'
Resque.enqueue(FileWorker, "12345567".split('').shuffle.join)
Rakefile (note the :environment task only makes sense if you are using Rails and will give errors otherwise):
require "resque/tasks"
require "./file_worker"
file_worker.rb:
class FileWorker
#queue = :save_to_file
def self.perform(str)
File.open('./' + Time.now.to_s + '.txt', 'w+') do |f|
f << "test 123"
end
end
end
Now the workers will be able to create instances of FileWorker to complete the tasks.
The way to avoid the warning about signals is given in the page linked to in the message. Simply set the environment variable TERM_CHILD when calling rake:
$ rake resque:work QUEUE='*' TERM_CHILD=1

Is Rails environment the prerequisite for cruisecontrol.rb

I have no rails environment but I want to use cruisecontrol.rb as my Continous Integration environment.
After following the instrcution from http://cruisecontrolrb.thoughtworks.com/documentation/getting_started and then
./cruise start
I got the error here: (sorry, but the formatter is better than posting it here directly)
http://pastebin.ca/1487868
It seems the CC.rb is doing some data migration/backup work when start up, and I could resolve this by comment out corresponding code :
#cruisecontrolrb / db / migrate / 002_move_custom_files_to_directory_in_user_home.rb
DATA_ROOT = ARGV[0]
RAILS_ROOT = File.expand_path(".")
if File.directory? 'projects'
#mv 'projects', DATA_ROOT + '/projects' #comment out this line, it will work perfect fine
else
mkdir_p DATA_ROOT + '/projects'
end
I debugged a litter bit and found when above code executing, the DATA_ROOT and Dir.pwd are ~/.cruise. So
mv 'projects', DATA_ROOT + '/projects' would become
mv ~/.cruise/projects ~/.cruise/projects which is obvious not correct
What would you recommend to solve this? To redfine DATA_ROOT to what even place I want?
There are several ways around this, the easiest is probably to create a cruise_config.rb file in the root of your project. It should look something like this :
Project.configure do |project|
project.rake_task = "spec"
end
just replace "spec" with whatever rake task you have. if you're not using rake (say you're using ant) you can instead do something like this :
Project.configure do |project|
project.build_command = "ant test"
end
just replace "ant test" with command line command that will return 0 if successful and 1 otherwise. (ant, make, rake, all do this)

Resources