Testing nested, asynchronous Sidekiq jobs with rspec - ruby

Lets say I have the following base Ruby class:
class BaseWorker
include Sidekiq::Worker
end
and the following two queues and one helper class:
class Finder
class << self
def find(id)
AwesomeDBClass.find(id)
end
end
end
class Queue1 extend BaseWorker
def perform(variable)
if variable.some_attribute
...
end
end
end
class Queue2 extend BaseWorker
def perform id
local_variable = Finder.find(id)
Queue1.perform_async(local_variable)
end
end
And the following test harness:
let(:local_variable) do
double(:local_var, some_attribute: true)
end
let(:local_variable2) do
OpenStruct(some_attribute: true)
end
before do
allow(Finder).to_receive(:find).and_return(:local_variable)
... code that triggers queue2 ...
queue2.drain
queue1.drain
end
This fails if local_variable (the double) is returned with:
undefined method `some_attribute' for {"__expired"=>false, "name"=>"local_variable"}:Hash
And if local_variable2 (the openstruct) is returned with:
undefined method `some_attribute' for {"table"=>{"some_attribute"=>true}}:Hash
interestingly, if inspecting either in queue2 (with a puts say) their values are correct.
Also, when reading the code for the two libraries:
https://github.com/ruby/ruby/blob/trunk/lib/ostruct.rb#L93
https://github.com/rspec/rspec-mocks/blob/31baac3ace4ec0aff97d9e4e1928b06d18c685f8/lib/rspec/mocks/test_double.rb#L10
It seems that the state suggest that the constructor has run, but some other thing (method_missing binding, eg other method definitions on the class and/or assign_stubs running) has not happened. Any ideas as to how or why this is happening would be appreciated.

TL;DR Use simple hashes instead of OpenStruct instances as params to perform.
Sidekiq serializes everything to Redis, please refer to Best Practices. OpenStruct is not a good candidate as a parameter to be passed to perform.
Minor sidenote: since Sidekiq overrides included callback, it would be more idiomatic to:
class BaseWorker
def self.extended base
base.send :include, Sidekiq::Worker
end
end

Related

Testing ruby superclass inherited functionality with rspec

Given that I have an abstract class which provides inherited functionality to subclasses:
class Superclass
class_attribute :_configuration_parameter
def self.configuration_parameter config
self._configuration_parameter = config
end
def results
unless #queried
execute
#queried = true
end
#results
end
private
# Execute uses the class instance config
def execute
#rows = DataSource.fetch self.class._configuration_parameter
#results = Results.new #rows, count
post_process
end
def post_process
#results.each do |row|
# mutate results
end
end
end
Which might be used by a subclass like this:
class Subclass < Superclass
configuration_parameter :foo
def subclass_method
end
end
I'm having a hard time writing RSpec to test the inherited and configured functionality without abusing the global namespace:
RSpec.describe Superclass do
let(:config_parameter) { :bar }
let(:test_subclass) do
# this feels like an anti-pattern, but the Class.new block scope
# doesn't contain config_parameter from the Rspec describe
$config_parameter = config_parameter
Class.new(Superclass) do
configuration_parameter $config_parameter
end
end
let(:test_instance) do
test_subclass.new
end
describe 'config parameter' do
it 'sets the class attribute' do
expect(test_subclass._configuration_parameter).to be(config_parameter)
end
end
describe 'execute' do
it 'fetches the data from the right place' do
expect(DataSource).to receive(:fetch).with(config_parameter)
instance.results
end
end
end
The real world superclass I'm mocking here has a few more configuration parameters and several other pieces of functionality which test reasonably well with this pattern.
Am I missing something obviously bad about the class or test design?
Thanks
I'm just going to jump to the most concrete part of your question, about how to avoid using a global variable to pass a local parameter to the dummy class instantiated in your spec.
Here's your spec code:
let(:test_subclass) do
# this feels like an anti-pattern, but the Class.new block scope
# doesn't contain config_parameter from the Rspec describe
$config_parameter = config_parameter
Class.new(Superclass) do
configuration_parameter $config_parameter
end
end
If you take the value returned from Class.new you can call configuration_parameter on that with the local value and avoid the global. Using tap does this with only a minor change to your existing code:
let(:test_subclass) do
Class.new(SuperClass).tap do |klass|
klass.configuration_parameter config_parameter
end
end
As to the more general question of how to test functionality inherited from a superclass, I think the general approach of creating a stub subclass and writing specs for that subclass is fine. I personally would make your _configuration_parameter class attribute private, and rather than testing that the configuration_parameter method actually sets the value, I'd instead focus on checking that the value is different from the superclass value. But I'm not sure that's in the scope of this question.

can common overriding code be shared

I am new to ruby and I'm hoping someone can help me. I am writing tests using test::unit and within my tests I needed to run some code before an assert was called so I overrided the assert methods like so:
class TestSomething < Test::Unit::TestCase
def assert_equal(expected, actual, message = nil)
mycode ..
super(expected, actual, message)
end
def assert(object, message)
my code ...
super(object, message)
end
def assert_not_nil(object, message = "")
my code ...
super(object, message)
end
def setup
end
def test_case1
end
def test_case1
end
def teardown
end
end
The above structure works fine and the asserts call my code. The thing is I have 100s of test classes. The assert overriding will be the same for all of them. Do I have to copy the assert overrides to the top of every class or is there a way for all of them to get the assert overrides in one go?
One more question. Is there a way of catching an error if it occurs anywhere within the entire class?
A
I am not sure why you do not want to use before filter, but the question as it was stated has an answer: since ruby classes are open, one might do the following.
class Test::Unit::TestCase
# store the original
alias_method :assert_equal_original, :assert_equal
# override
def assert_equal *args
# mycode ..
assert_equal_original *args
end
# the same for other methods
end
Once it is done, any derived class will call my_code before calling the original method.
To catch an error within the scope of the class is impossible, AFAIK.

Having more than one method in a sidekiq worker class

I have a sidekiq worker class. I currently implemented it this way. It works when I call PROCESS, and it will queue the method called PERFORM. But i would like to have more than one method that I can queue.
As a side note, is there a difference doing this and simply doing SocialSharer.delay.perform?
# I trigger by using SocialSharer.process("xxx")
class SocialSharer
include Sidekiq::Worker
def perform(user_id)
# does things
end
def perform_other_things
#i do not know how to trigger this
end
class << self
def process(user_id)
Sidekiq::Client.enqueue(SocialSharer,user_id)
end
end
end
SocialSharer.delay.perform would delay a class method called perform. Your perform method is an instance method.
Workers are designed to be one class per job. The job is started via the perform method. You can use delay to kick off any number of different class methods on a class, like so:
class Foo
def self.a(count)
end
def self.b(name)
end
end
Foo.delay.a(10)
Foo.delay.b('bob')
Well, if you really want to have all "performable" methods in one class, I'd suggest you to rename perform method to something different (say, perform_something), and create a new perform method which would dispatch the control flow:
class SocialSharer
include Sidekiq::Worker
# the 3 lines below may be replaced with `alias_method :perform, :public_send`
def perform(method, *args)
self.public_send(method, *args)
end
def perform_something(user_id)
# does things
end
def perform_other_things
# does other things
end
def self.process(user_id)
Sidekiq::Client.enqueue(SocialSharer, :perform_something, user_id)
Sidekiq::Client.enqueue(SocialSharer, :perform_other_things)
end
end

Ruby Static method with local scope

The title sounds rediculous because it is. My biggest issue is actually trying to figure out what question to ask.
The goal: To be able to implement the code as described below OR to figure out what terminology I should be using to search for the correct answer.
The issue: I wish to have a system where classes register "processors" via a method within the class definition. eg:
class RunTheseMethodsWhenICallProcess
Include ProcessRunner
add_processor :a_method_to_run
add_processor :another_method_to_run
def a_method_to_run
puts "This method ran"
end
def another_method_to_run
puts "another method ran"
end
end
Module ProcessRunner
def process
processors.each {|meth| self.send(meth)}
end
end
My issues are mostly with understanding the scope and reference of the class to make them interact. As it stands, I have been able to add a static method 'add_processor' by calling class.extend(AClass) in the included method and adding in the class there.
The idea for this syntax was inspired by DataMappers 'property' and 'before' methods. Even with the code checked out, I am having a touch of trouble following it.
Thanks so much for any help you can offer.
If I got you right, the following will do what you want.
It initializes each class (or module) including ProcessRunner to have an empty array in ##processors. Additionally it adds class methods processors (a simple getter) and add_processor.
The process method had to be adjusted to use the class method. In fact, you could add a wrapper for this, but I think that would be to verbose for such a sample.
module ProcessRunner
module ClassMethods
def add_processor(processor)
processors << processor
end
def processors
class_variable_get :##processors
end
end
def self.included(mod)
mod.send :class_variable_set, :##processors, []
mod.extend ClassMethods
end
def process
self.class.processors.each {|meth| self.send(meth)}
end
end
class RunTheseMethodsWhenICallProcess
include ProcessRunner
add_processor :a_method_to_run
add_processor :another_method_to_run
def a_method_to_run
puts "This method ran"
end
def another_method_to_run
puts "another method ran"
end
end

Ruby design pattern: How to make an extensible factory class?

Ok, suppose I have Ruby program to read version control log files and do something with the data. (I don't, but the situation is analogous, and I have fun with these analogies). Let's suppose right now I want to support Bazaar and Git. Let's suppose the program will be executed with some kind of argument indicating which version control software is being used.
Given this, I want to make a LogFileReaderFactory which given the name of a version control program will return an appropriate log file reader (subclassed from a generic) to read the log file and spit out a canonical internal representation. So, of course, I can make BazaarLogFileReader and GitLogFileReader and hard-code them into the program, but I want it to be set up in such a way that adding support for a new version control program is as simple as plopping a new class file in the directory with the Bazaar and Git readers.
So, right now you can call "do-something-with-the-log --software git" and "do-something-with-the-log --software bazaar" because there are log readers for those. What I want is for it to be possible to simply add a SVNLogFileReader class and file to the same directory and automatically be able to call "do-something-with-the-log --software svn" without ANY changes to the rest of the program. (The files can of course be named with a specific pattern and globbed in the require call.)
I know this can be done in Ruby... I just don't how I should do it... or if I should do it at all.
You don't need a LogFileReaderFactory; just teach your LogFileReader class how to instantiate its subclasses:
class LogFileReader
def self.create type
case type
when :git
GitLogFileReader.new
when :bzr
BzrLogFileReader.new
else
raise "Bad log file type: #{type}"
end
end
end
class GitLogFileReader < LogFileReader
def display
puts "I'm a git log file reader!"
end
end
class BzrLogFileReader < LogFileReader
def display
puts "A bzr log file reader..."
end
end
As you can see, the superclass can act as its own factory. Now, how about automatic registration? Well, why don't we just keep a hash of our registered subclasses, and register each one when we define them:
class LogFileReader
##subclasses = { }
def self.create type
c = ##subclasses[type]
if c
c.new
else
raise "Bad log file type: #{type}"
end
end
def self.register_reader name
##subclasses[name] = self
end
end
class GitLogFileReader < LogFileReader
def display
puts "I'm a git log file reader!"
end
register_reader :git
end
class BzrLogFileReader < LogFileReader
def display
puts "A bzr log file reader..."
end
register_reader :bzr
end
LogFileReader.create(:git).display
LogFileReader.create(:bzr).display
class SvnLogFileReader < LogFileReader
def display
puts "Subersion reader, at your service."
end
register_reader :svn
end
LogFileReader.create(:svn).display
And there you have it. Just split that up into a few files, and require them appropriately.
You should read Peter Norvig's Design Patterns in Dynamic Languages if you're interested in this sort of thing. He demonstrates how many design patterns are actually working around restrictions or inadequacies in your programming language; and with a sufficiently powerful and flexible language, you don't really need a design pattern, you just implement what you want to do. He uses Dylan and Common Lisp for examples, but many of his points are relevant to Ruby as well.
You might also want to take a look at Why's Poignant Guide to Ruby, particularly chapters 5 and 6, though only if you can deal with surrealist technical writing.
edit: Riffing of off Jörg's answer now; I do like reducing repetition, and so not repeating the name of the version control system in both the class and the registration. Adding the following to my second example will allow you to write much simpler class definitions while still being pretty simple and easy to understand.
def log_file_reader name, superclass=LogFileReader, &block
Class.new(superclass, &block).register_reader(name)
end
log_file_reader :git do
def display
puts "I'm a git log file reader!"
end
end
log_file_reader :bzr do
def display
puts "A bzr log file reader..."
end
end
Of course, in production code, you may want to actually name those classes, by generating a constant definition based on the name passed in, for better error messages.
def log_file_reader name, superclass=LogFileReader, &block
c = Class.new(superclass, &block)
c.register_reader(name)
Object.const_set("#{name.to_s.capitalize}LogFileReader", c)
end
This is really just riffing off Brian Campbell's solution. If you like this, please upvote his answer, too: he did all the work.
#!/usr/bin/env ruby
class Object; def eigenclass; class << self; self end end end
module LogFileReader
class LogFileReaderNotFoundError < NameError; end
class << self
def create type
(self[type] ||= const_get("#{type.to_s.capitalize}LogFileReader")).new
rescue NameError => e
raise LogFileReaderNotFoundError, "Bad log file type: #{type}" if e.class == NameError && e.message =~ /[^: ]LogFileReader/
raise
end
def []=(type, klass)
#readers ||= {type => klass}
def []=(type, klass)
#readers[type] = klass
end
klass
end
def [](type)
#readers ||= {}
def [](type)
#readers[type]
end
nil
end
def included klass
self[klass.name[/[[:upper:]][[:lower:]]*/].downcase.to_sym] = klass if klass.is_a? Class
end
end
end
def LogFileReader type
Here, we create a global method (more like a procedure, actually) called LogFileReader, which is the same name as our module LogFileReader. This is legal in Ruby. The ambiguity is resolved like this: the module will always be preferred, except when it's obviously a method call, i.e. you either put parentheses at the end (Foo()) or pass an argument (Foo :bar).
This is a trick that is used in a few places in the stdlib, and also in Camping and other frameworks. Because things like include or extend aren't actually keywords, but ordinary methods that take ordinary parameters, you don't have to pass them an actual Module as an argument, you can also pass anything that evaluates to a Module. In fact, this even works for inheritance, it is perfectly legal to write class Foo < some_method_that_returns_a_class(:some, :params).
With this trick, you can make it look like you are inheriting from a generic class, even though Ruby doesn't have generics. It's used for example in the delegation library, where you do something like class MyFoo < SimpleDelegator(Foo), and what happens, is that the SimpleDelegator method dynamically creates and returns an anonymous subclass of the SimpleDelegator class, which delegates all method calls to an instance of the Foo class.
We use a similar trick here: we are going to dynamically create a Module, which, when it is mixed into a class, will automatically register that class with the LogFileReader registry.
LogFileReader.const_set type.to_s.capitalize, Module.new {
There's a lot going on in just this line. Let's start from the right: Module.new creates a new anonymous module. The block passed to it, becomes the body of the module – it's basically the same as using the module keyword.
Now, on to const_set. It's a method for setting a constant. So, it's the same as saying FOO = :bar, except that we can pass in the name of the constant as a parameter, instead of having to know it in advance. Since we are calling the method on the LogFileReader module, the constant will be defined inside that namespace, IOW it will be named LogFileReader::Something.
So, what is the name of the constant? Well, it's the type argument passed into the method, capitalized. So, when I pass in :cvs, the resulting constant will be LogFileParser::Cvs.
And what do we set the constant to? To our newly created anonymous module, which is now no longer anonymous!
All of this is really just a longwinded way of saying module LogFileReader::Cvs, except that we didn't know the "Cvs" part in advance, and thus couldn't have written it that way.
eigenclass.send :define_method, :included do |klass|
This is the body of our module. Here, we use define_method to dynamically define a method called included. And we don't actually define the method on the module itself, but on the module's eigenclass (via a small helper method that we defined above), which means that the method will not become an instance method, but rather a "static" method (in Java/.NET terms).
included is actually a special hook method, that gets called by the Ruby runtime, everytime a module gets included into a class, and the class gets passed in as an argument. So, our newly created module now has a hook method that will inform it whenever it gets included somewhere.
LogFileReader[type] = klass
And this is what our hook method does: it registers the class that gets passed into the hook method into the LogFileReader registry. And the key that it registers it under, is the type argument from the LogFileReader method way above, which, thanks to the magic of closures, is actually accessible inside the included method.
end
include LogFileReader
And last but not least, we include the LogFileReader module in the anonymous module. [Note: I forgot this line in the original example.]
}
end
class GitLogFileReader
def display
puts "I'm a git log file reader!"
end
end
class BzrFrobnicator
include LogFileReader
def display
puts "A bzr log file reader..."
end
end
LogFileReader.create(:git).display
LogFileReader.create(:bzr).display
class NameThatDoesntFitThePattern
include LogFileReader(:darcs)
def display
puts "Darcs reader, lazily evaluating your pure functions."
end
end
LogFileReader.create(:darcs).display
puts 'Here you can see, how the LogFileReader::Darcs module ended up in the inheritance chain:'
p LogFileReader.create(:darcs).class.ancestors
puts 'Here you can see, how all the lookups ended up getting cached in the registry:'
p LogFileReader.send :instance_variable_get, :#readers
puts 'And this is what happens, when you try instantiating a non-existent reader:'
LogFileReader.create(:gobbledigook)
This new expanded version allows three different ways of defining LogFileReaders:
All classes whose name matches the pattern <Name>LogFileReader will automatically be found and registered as a LogFileReader for :name (see: GitLogFileReader),
All classes that mix in the LogFileReader module and whose name matches the pattern <Name>Whatever will be registered for the :name handler (see: BzrFrobnicator) and
All classes that mix in the LogFileReader(:name) module, will be registered for the :name handler, regardless of their name (see: NameThatDoesntFitThePattern).
Please note that this is just a very contrived demonstration. It is, for example, definitely not thread-safe. It might also leak memory. Use with caution!
One more minor suggestion for Brian Cambell's answer -
In you can actually auto-register the subclasses with an inherited callback. I.e.
class LogFileReader
cattr_accessor :subclasses; self.subclasses = {}
def self.inherited(klass)
# turns SvnLogFileReader in to :svn
key = klass.to_s.gsub(Regexp.new(Regexp.new(self.to_s)),'').underscore.to_sym
# self in this context is always LogFileReader
self.subclasses[key] = klass
end
def self.create(type)
return self.subclasses[type.to_sym].new if self.subclasses[type.to_sym]
raise "No such type #{type}"
end
end
Now we have
class SvnLogFileReader < LogFileReader
def display
# do stuff here
end
end
With no need to register it
This should work too, without the need for registering class names
class LogFileReader
def self.create(name)
classified_name = name.to_s.split('_').collect!{ |w| w.capitalize }.join
Object.const_get(classified_name).new
end
end
class GitLogFileReader < LogFileReader
def display
puts "I'm a git log file reader!"
end
end
and now
LogFileReader.create(:git_log_file_reader).display
This is how I would make an extensible factory class.
module Factory
class Error < RuntimeError
end
class Base
##registry = {}
class << self
def inherited(klass)
type = klass.name.downcase.to_sym
##registry[type] = klass
end
def create(type, *args, **kwargs)
klass = ##registry[type]
return klass.new(*args, **kwargs) if klass
raise Factory::Error.new "#{type} is unknown"
end
end
end
end
class Animal < Factory::Base
attr_accessor :name
def initialize(name)
#name = name
end
def walk?
raise NotImplementedError
end
end
class Cat < Animal
def walk?; true; end
end
class Fish < Animal
def walk?; false; end
end
class Salmon < Fish
end
duck = Animal.create(:cat, "Garfield")
salmon = Animal.create(:salmon, "Alfredo")
pixou = Animal.create(:duck, "Pixou") # duck is unknown (Factory::Error)

Resources