I've tried using Resque before and was met with unmitigated failure. I'm revisiting it again with the same results...
resque.rake:
require "resque/tasks"
task "resque:setup" => :environment
test.rb:
require 'resque'
class FileWorker
#queue = :save_to_file
def self.perform(str)
File.open('./' + Time.now.to_s + '.txt', 'w+') do |f|
f << "test 123"
end
end
end
Resque.enqueue(FileWorker, "12345567".split('').shuffle.join)
Gemfile:
gem 'resque'
gem 'rake'
It seems like running test.rb on its own successfully queues the job:
However, running rake resque:work QUEUE='*' in the same folder results in a warning,
WARNING: This way of doing signal handling is now deprecated. Please
see http://hone.heroku.com/resque/2012/08/21/resque-signals.html for
more info.
As well as the task being added to "failed" queue with the following reason: "exception":"NameError","error":"uninitialized constant FileWorker"
How do I get this to work? Seems like something quite obvious but there's tons of tutorials about Resque spanning many years - some painfully out of date and none explaining how to run workers so they don't fail.
Thanks in advance.
When you enqueue a task with Resque, what is stored on Redis is just the name of the job class (as a string) along with the arguments (again as strings) in a JSON object.
When a worker then tries to perform the task, it needs to be able to create an instance of the job class. It does this by using const_get and const_missing. This is where the error you are seeing occurs, since the worker does not have the definition of FileWorker available to it.
The error is the same as if you tried to get an unknown constant in irb:
> Object.const_missing "FileWorker"
NameError: uninitialized constant FileWorker
The solution is to make sure the definition of FileWorker is available to your workers. The simplest way to do this would be to just require test.rb from your Rakefile (or resque.rake). In your code this would involve adding another task to the queue, so you might want to move the FileWorker code to its own file where it can be required by both the rake file and the code enqueuing jobs.
test.rb:
require 'resque'
require './file_worker'
Resque.enqueue(FileWorker, "12345567".split('').shuffle.join)
Rakefile (note the :environment task only makes sense if you are using Rails and will give errors otherwise):
require "resque/tasks"
require "./file_worker"
file_worker.rb:
class FileWorker
#queue = :save_to_file
def self.perform(str)
File.open('./' + Time.now.to_s + '.txt', 'w+') do |f|
f << "test 123"
end
end
end
Now the workers will be able to create instances of FileWorker to complete the tasks.
The way to avoid the warning about signals is given in the page linked to in the message. Simply set the environment variable TERM_CHILD when calling rake:
$ rake resque:work QUEUE='*' TERM_CHILD=1
Related
I'm using ruby-prolog. I want to run a task to query a fact.
demo.rb:
require 'ruby-prolog'
c = RubyProlog::Core.new
c.instance_eval do
person['name','brian'].fact
person['name','James'].fact
puts 'all the names are: '
p query(person['name', :A])
end
This works great. Now I want to run the query inside of Rake. That is a problem because I don't know how to access person[] from the other file.
Rakefile.rb:
require_relative 'demo.rb'
task :test do |variable|
puts 'all the names are: '
p query(person['name', :A])
end
Error:
all the names are: rake aborted! NameError: undefined local variable
or method `person' for main:Object
I'm hoping this can be solved by passing an object back somehow. I tried accessing c, but it did not work out. Any ideas?
In your demo file, both variables person and c are local variables and will not be accessible from outside of that context. If you require demo.rb into an irb session, the behavior should be the same; neither c nor person will be defined.
A good way of dealing with this in rake tasks is to keep any and all logic out of the rake task itself, and only call out to another object that takes care of the task. For a quick and dirty example, you could alter your code as such:
# demo.rb
require 'ruby-prolog'
class Demo
def self.run_demo
# Existing code:
c = RubyProlog::Core.new
c.instance_eval do
person['name','brian'].fact
person['name','James'].fact
puts 'all the names are: '
p query(person['name', :A])
end
end
end
and
# Rakefile.rb
require_relative 'demo.rb'
task :test do
Demo.run_demo
end
I'm using Sidekiq 3.1.2 without Rails like this:
$ sidekiq -vr sidekiq.rb
sidekiq.rb looks like this:
($LOAD_PATH << '.' << 'lib' << 'lib/workers').uniq!
require 'lookup_worker'
lib/workers/lookup_worker.rb looks like this:
require 'sidekiq'
class LookupWorker
include Sidekiq::Worker
def perform(*args)
puts "LookupWorker#perform fired with arguments #{args.map(&:inspect).join(', ')}"
end
end
But when I'm in irb and try
> LookupWorker.perform_async('asdf')
it gives me this:
WARN: {"retry"=>true, "queue"=>"default", "class"=>"LookupWorker", "args"=>["asdf"], "jid"=>"8c278868c5f05ec9beb1dbae", "enqueued_at"=>1402457226.9612548}
WARN: uninitialized constant LookupWorker
WARN: [backtrace, none of it from my code]
ERROR: Sidekiq::Processor crashed!
NameError: uninitialized constant LookupWorker
What am I missing?
So...
In LookupWorker, require was getting confused between sidekiq the gem and sidekiq the script on line 1.
My sidekiq.rb needed to be renamed as sidekiq_script.rb (or anything else really). Only gotcha is, I have to include the directory when running sidekiq:
$ sidekiq -r ./sidekiq_script.rb
not
$ sidekiq -r sidekiq_script.rb
Well I feel slightly stupid for that.
Can I conditionally skip loading "further" ruby code in the same file,
if a library (loaded via require) is not found ?
begin
require 'aws-sdk'
rescue LoadError
puts "aws-sdk gem not found"
return #does not work. nor does next
end
# code after here should not be executed as `aws-sdk` gem was not found
puts "=== should not get executed"
namespace :db do
desc "import local postgres database to heroku. user and database name is hardcoded"
task :import_to_heroku => [:environment, "db:dump_for_heroku"] do
# code using aws-sdk gem
end
end
In the above code, can I ask Ruby not to load the file further after
hitting a rescue LoadError.
Like an early return but for loading a file and not for a function.
Need it because i have i have a rake task which needs aws-sdk rubygem but i use it only
on my local machine. If aws-sdk is not found it does not make sense for me to load code afterwards in the same file. I guess i can split the code into smaller files and warp it in
a require call
if Rails.env.development?
require 'import_to_heroku'
end
But do not want to warp or modify my existing code
Also, i can wrap the whole code in an conditional but that is inelegant.
A begin-rescue block is also a form of explicit control flow.
I do not want to wrap or touch the original code is any manner
Maybe an api such as
require_or_skip_further_loading 'aws-ruby`
So i want my code to be functionally equivalent to
begin
require 'aws-sdk'
namespace :db do
desc "import local postgres database to heroku. user and database name is hardcoded"
task :import_to_heroku => [:environment, "db:dump_for_heroku"] do
# code using aws-sdk gem
end
end
rescue LoadError
puts "aws-sdk gem not found"
end
Or via an if conditional
library_found = false
begin
require 'aws-sdk'
library_found = true
rescue LoadError
puts "aws-sdk gem not found"
return #does not work
end
if library_found
namespace :db do
desc "import local postgres database to heroku. user and database name is hardcoded"
task :import_to_heroku => [:environment, "db:dump_for_heroku"] do
# code using aws-sdk gem
end
end
end
Want program execution to continue after LoadError is raised. ie. gracefully handle LoadError and do not load code written after LoadError in the same file. cannot raise exit or abort on LoadError And particularly the code after LoadError should not be executed (or loaded) by the ruby interpreter
Had originally asked How to skip require in ruby?
but i did not ask the question properly. hope this is better worded
I haven't checked the source, but I suppose that, when you run ruby my_file.rb at the console, or require/load it from Ruby code, the file is read entirely into memory before being evaluated. I'm afraid there is no such thing as skipping a part of a file.
I had an idea with catch/throw.
requiring file (for example a Rake task ?) treq1.rb :
catch :aws_sdk do
require_relative 'original'
end
puts '... continued'
The original file that you don't want to modify, original.rb :
puts 'in original, not to be modified'
begin
require 'aws-sdk'
rescue LoadError
puts "aws-sdk gem not found"
throw :aws_sdk
end
puts ">>> to execute only if 'aws-sdk' is found"
# namespace :db do ... etc
#end
Execution :
$ ruby -w treq1.rb
in original, not to be modified
aws-sdk gem not found
treq1.rb:2:in `require_relative': method `backtrace' called on unexpected T_NODE object (0x007fd32b88e900 flags=0x381c klass=0x0) (NotImplementedError)
from treq1.rb:2:in `block in <main>'
from treq1.rb:1:in `catch'
from treq1.rb:1:in `<main>'
Googling with the error : http://www.ruby-forum.com/topic/4406870
Recent post, no answer. It works in a single file, if you want to wrap your code.
Let's try another solution. Supposing that you can change the Rake task, treq2.rb :
begin
require_relative 'original'
rescue LocalJumpError
puts 'rescued LocalJumpError'
end
puts '... continued'
In original.rb, replace throw :aws_sdk by return :
$ ruby -w treq2.rb
in original, not to be modified
aws-sdk gem not found
rescued LocalJumpError
... continued
This way it works.
HTH
The "top-level return" feature has been added.
It is now possible to use the return keyword at the top level, as in your "via an if conditional" example. Further discussion here.
Are the following equivalent?
namespace :resque do
task setup: :environment do
end
end
task "resque:setup" => :environment do
end
In short: yes. When running rake resque:setup both of these tasks will be invoked.
Rake will merge these tasks. You can test this by doing the following:
p Rake.application.tasks
Which in this case would return something like
[<Rake::Task resque:setup => [environment]>]
Which is simply an Array holding a single Rake::Task object. You can also check the scope or list of namespaces for a task by doing:
p Rake.application.tasks.first.scope
#=> ["resque"]
If you want to learn a little more on how the internals of Rake work, check out Rake::Task and Rake::TaskManager
I have a Rakefile with a Rake task that I would normally call from the command line:
rake blog:post Title
I'd like to write a Ruby script that calls that Rake task multiple times, but the only solution I see is shelling out using `` (backticks) or system.
What's the right way to do this?
from timocracy.com:
require 'rake'
def capture_stdout
s = StringIO.new
oldstdout = $stdout
$stdout = s
yield
s.string
ensure
$stdout = oldstdout
end
Rake.application.rake_require 'metric_fetcher', ['../../lib/tasks']
results = capture_stdout {Rake.application['metric_fetcher'].invoke}
This works with Rake version 10.0.3:
require 'rake'
app = Rake.application
app.init
# do this as many times as needed
app.add_import 'some/other/file.rake'
# this loads the Rakefile and other imports
app.load_rakefile
app['sometask'].invoke
As knut said, use reenable if you want to invoke multiple times.
You can use invoke and reenable to execute the task a second time.
Your example call rake blog:post Title seems to have a parameter. This parameter can be used as a parameter in invoke:
Example:
require 'rake'
task 'mytask', :title do |tsk, args|
p "called #{tsk} (#{args[:title]})"
end
Rake.application['mytask'].invoke('one')
Rake.application['mytask'].reenable
Rake.application['mytask'].invoke('two')
Please replace mytask with blog:post and instead the task definition you can require your rakefile.
This solution will write the result to stdout - but you did not mention, that you want to suppress output.
Interesting experiment:
You can call the reenable also inside the task definition. This allows a task to reenable himself.
Example:
require 'rake'
task 'mytask', :title do |tsk, args|
p "called #{tsk} (#{args[:title]})"
tsk.reenable #<-- HERE
end
Rake.application['mytask'].invoke('one')
Rake.application['mytask'].invoke('two')
The result (tested with rake 10.4.2):
"called mytask (one)"
"called mytask (two)"
In a script with Rails loaded (e.g. rails runner script.rb)
def rake(*tasks)
tasks.each do |task|
Rake.application[task].tap(&:invoke).tap(&:reenable)
end
end
rake('db:migrate', 'cache:clear', 'cache:warmup')