Inheritance within iron workers when using the iron_worker_ruby gem - ruby

I'm considering using IronWorker a project so that I can scale it easily (high traffic expected, with lots of background jobs).
In order to stay DRY, I'm trying to define workers using inheritance but I keep getting the following error:
/usr/local/lib/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- base_worker.rb (LoadError)
from /usr/local/lib/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /task/child_worker.rb:3:in `<top (required)>'
from /task/runner.rb:344:in `require_relative'
from /task/runner.rb:344:in `<main>'
Here is the base worker class:
# app/workers/base_worker.rb
require 'net/http'
require 'uri'
require 'json'
class BaseWorker < IronWorker::Base
attr_accessor :params
# The run method is what IronWorker calls to run your worker
def run
data = custom_run(params)
common_post_process(data)
end
def custom_run(params)
#to be overwritten in the child class
end
def common_post_process(data)
# some common post processing => DRY
....
end
end
And here is a child class :
# app/workers/child_worker.rb
require 'net/http'
require 'uri'
require 'base_worker.rb'
class ChildWorker < BaseWorker
merge "base_worker.rb"
def custom_run(params)
#custom work
end
end
Any idea on how to fix this?

I'd recommend using our next generation gem, iron_worker_ng: https://github.com/iron-io/iron_worker_ruby_ng . The iron_worker gem is deprecated. And if you want to keep it similar style to what you have, your child_worker.rb might look like this:
require 'net/http'
require 'uri'
require_relative 'base_worker.rb'
class ChildWorker < BaseWorker
def custom_run(params)
#custom work
end
end
# NG gem doesn't run anything in particular, so to run your method:
cw = ChildWorker.new
cw.custom_run(params)
And in a child_worker.worker file:
runtime 'ruby'
file 'base_worker.rb'
exec 'child_worker.rb'
Then to upload it to IronWorker:
iron_worker upload child_worker
Then you can start queuing jobs for it:
worker = IronWorkerNG::Client.new
worker.tasks.create("child_worker", params)

If you use iron_worker_ng, it also possible to define a run method. This method will be called when the IronWorker runs. You have to specify the Class within the .worker file.
# child_worker.rb
class ChildWorker
def run
puts "doing the hard work"
end
end
And the child_worker.worker file:
# child_worker.worker
runtime 'ruby'
name 'ChildWorker'
exec 'child_worker.rb', 'ChildWorker'

Related

How do I reference a method in a different class from a method in another class?

I have a module and class in a file lib/crawler/page-crawler.rb that looks like this:
require 'oga'
require 'net/http'
require 'pry'
module YPCrawler
class PageCrawler
attr_accessor :url
def initialize(url)
#url = url
end
def get_page_listings
body = Net::HTTP.get(URI.parse(#url))
document = Oga.parse_html(body)
document.css('div.result')
end
newpage = PageCrawler.new "http://www.someurl"
#listings = newpage.get_page_listings
#listings.each do |listing|
bizname = YPCrawler::ListingCrawler.new listing['id']
end
end
end
Then I have another module & class in another file lib/crawler/listing-crawler.rb that looks like this:
require 'oga'
require 'pry'
module YPCrawler
class ListingCrawler
def initialize(id)
#id = id
end
def extract_busines_name
binding.pry
end
end
end
However, when I try to run this script ruby lib/yp-crawler.rb which executes the page-crawler.rb file above and works without the YPCrawler call, I get this error:
/lib/crawler/page-crawler.rb:23:in `block in <class:PageCrawler>': uninitialized constant YPCrawler::ListingCrawler (NameError)
The issue is on this line:
bizname = YPCrawler::ListingCrawler.new listing['id']
So how do I call that other from within my iterator in my page-crawler.rb?
Edit 1
When I just do `ListingCrawler.new listing['id'], I get the following error:
uninitialized constant YPCrawler::PageCrawler::ListingCrawler (NameError)
Edit 2
Here is the directory structure of my project:
Edit 3
My yp-crawler.rb looks like this:
require_relative "yp-crawler/version"
require_relative "crawler/page-crawler"
require_relative "crawler/listing-crawler"
module YPCrawler
end
In your yp-crawler.rb file, based on the structure that you posted, you should have something like:
require 'yp-crawler/version'
require 'crawler/listing-crawler'
require 'crawler/page-crawler'
Try this, in your yp-crawler.rb add the line:
Dir["#{File.dirname(__FILE__)}/crawler/**/*.rb"].each { |file| load(file) }
That should automatically include all files in your /crawler directory at runtime. Might want to do the same for the other directories.
Let me know if that helps :)

Set Up for RSpec in a Sinatra modular app

This is my first attempt with Sinatra. I built a simple classic app, set up RSpec for it, and got it working. Then, I tried to go modular, in a MVC fashion. Even though the app works in the browser, RSpec throws a NoMethodError. I've read Sinatra docs regarding RSpec, also searched a lot here in SO, but I can't find where the bug is. Any clue?
Thank you very much in advance.
Here are my relevant files:
config.ru
require 'sinatra/base'
Dir.glob('./{app/controllers}/*.rb') { |file| require file }
map('/') { run ApplicationController }
app.rb
require 'sinatra/base'
class ZerifApp < Sinatra::Base
# Only start the server if this file has been
# executed directly
run! if __FILE__ == $0
end
app/controllers/application_controller.rb
class ApplicationController < Sinatra::Base
set :views, File.expand_path('../../views', __FILE__)
set :public_dir, File.expand_path('../../../public', __FILE__)
get '/' do
erb :index
end
end
spec/spec_helper.rb
require 'rack/test'
# Also tried this
# Rack::Builder.parse_file(File.expand_path('../../config.ru', __FILE__))
require File.expand_path '../../app.rb', __FILE__
ENV['RACK_ENV'] = 'test'
module RSpecMixin
include Rack::Test::Methods
def app() described_class end
end
RSpec.configure { |c| c.include RSpecMixin }
spec/app_spec.rb
require File.expand_path '../spec_helper.rb', __FILE__
describe "My Sinatra Application" do
it "should allow accessing the home page" do
get '/'
expect(last_response).to be_ok
end
end
The error
My Sinatra Application should allow accessing the home page
Failure/Error: get '/'
NoMethodError:
undefined method `call' for nil:NilClass
# ./spec/app_spec.rb:5:in `block (2 levels) in <top (required)>'
I'm guessing you're following this recipe, correct?
The described_class in this line:
def app() described_class end
is meant to be the class under test, in this case ZerifApp. Try it like so:
def app() ZerifApp end
EDIT
It turns out the above answer is not correct about what described_class does. I assumed it was a placeholder -- actually it is an RSpec method that returns the class of the implicit subject, that is to say, the thing being tested.
The recipe at the link is misleading because of the way it recommends writing the describe block:
describe "My Sinatra Application" do
This is valid RSpec, but it does not define the subject class. Executing described_class in an example for this block will return nil. To make it work, replace the describe block:
describe ZerifApp do
Now described_class will return the expected value (ZerifApp)
https://pragprog.com/book/7web/seven-web-frameworks-in-seven-weeks
It has some source code to get some ideas from.
This has code example too. https://github.com/laser/sinatra-best-practices

'Error: Cannot open "/home/<...>/billy-bones/=" for reading' while using pry and DataMapper

So, I'm trying to build a quick console program for my development needs, akin to rails console (I'm using Sinatra + DataMapper + pry).
I run it and launch cat = Category.new(name: 'TestCat', type: :referential). It gives me the following error:
Error: Cannot open "/home/art-solopov/Projects/by-language/Ruby/billy-bones/=" for reading.
What could be the cause of the problem?
console:
#!/usr/bin/env ruby
$LOAD_PATH << 'lib'
require 'pry'
require 'config'
binding.pry
lib/config.rb:
# Configuration files and app-wide requires go here
require 'sinatra'
require 'data_mapper'
require 'model/bill'
require 'model/category'
configure :production do
DataMapper::Logger.new('db-log', :debug)
DataMapper.setup(:default,
'postgres://billy-bones:billy#localhost/billy-bones')
DataMapper.finalize
end
configure :development do
DataMapper::Logger.new($stderr, :debug)
DataMapper.setup(:default,
'postgres://billy-bones:billy#localhost/billy-bones-dev')
DataMapper.finalize
DataMapper.auto_upgrade!
end
configure :test do
require 'dm_migrations'
DataMapper::Logger.new($stderr, :debug)
DataMapper.setup(:default,
'postgres://billy-bones:billy#localhost/billy-bones-test')
DataMapper.finalize
DataMapper.auto_migrate!
end
lib/model/category.rb:
require 'data_mapper'
class Category
include DataMapper::Resource
property :id, Serial
property :name, String
property :type, Enum[:referential, :predefined, :computable]
has n, :bills
# has n, :tariffs TODO uncomment when tariff ready
def create_bill(params)
# A bill factory for current category type
case type
when :referential
ReferentialBill.new params
when :predefined
PredefinedBill.new params
when :computable
ComputableBill.new params
end
end
end
If I substitute pry with irb in the console script, it goes fine.
Thank you very much!
P. S.
Okay, yesterday I tried this script again, and it worked perfectly. I didn't change anything. I'm not sure whether I should remove the question now or not.
P. P. S.
Or actually not... Today I've encountered it again. Still completely oblivious to what could cause it.
** SOLVED **
DAMN YOU PRY!
Okay, so here's the difference.
When I tested it the second time, I actually entered a = Category.new(name: 'TestCat', type: :referential) and it worked. Looks like pry just thinks cat is a Unix command, not a valid variable name.
Not answer to the pry question I just generally hate case statements in ruby.
Why not change:
def create_bill(params)
# A bill factory for current category type
case type
when :referential
ReferentialBill.new params
when :predefined
PredefinedBill.new params
when :computable
ComputableBill.new params
end
end
to:
def create_bill(params)
# A bill factory for current category type
self.send("new_#{type}_bill",params)
end
def new_referential_bill(params)
ReferentialBill.new params
end
def new_predefined_bill(params)
PredefinedBill.new params
end
def new_computable_bill(params)
ComputableBill.new params
end
You could make this more dynamic but I think that would take away from readability in this case but if you'd like in rails this should do the trick
def create_bill(params)
if [:referential, :predefined, :computable].include?(type)
"#{type}_bill".classify.constantize.new(params)
else
#Some Kind of Handling for non Defined Bill Types
end
end
Or this will work inside or outside rails
def create_bill(params)
if [:referential, :predefined, :computable].include?(type)
Object.const_get("#{type.to_s.capitalize}Bill").new(params)
else
#Some Kind of Handling for non Defined Bill Types
end
end

uninitialized constant BikeShare (NameError)

I'm trying to implement some simple testing in rspec for a gem I'm writing. When I comment out describe BikeShare do down to end and run the file, the file loads in and runs successfully. I'm sure it's something tiny I'm missing.
My test file is really simple and looks like this:
require 'spec_helper'
describe BikeShare do
it "should run" do
# response = BikeShare.new
# response.should_be present
end
end
When run, I get the error uninitialized constant BikeShare (NameError) at line 3.
My bikeshare.rb file looks like this, fairly simple:
class BikeShare
def initialize
response = JSON.parse(open("http://bayareabikeshare.com/stations/json").read)
#response = response["stationBeanList"]
end
def get_last_station
#response.last["id"]
end
end
My Rakefile looks like this:
require 'rubygems'
require 'bundler'
Bundler.setup
Bundler::GemHelper.install_tasks
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new do |spec|
# spec.libs << 'lib' << 'spec'
spec.pattern = 'spec/*_spec.rb'
end
task :default => :spec
Your tests arent aware of BikeShare.
You need to require the file that defines your BikeShare class. I dont use rspec but I think that you normally set up your testing environment in spec_helper.rb.

Webrat Mechanize outside of Rails

I'm trying to use Webrat in a standalone script to automate some web browsing. How do I get the assert_contain method to work?
require 'rubygems'
require 'webrat'
include Webrat::Methods
include Webrat::Matchers
Webrat.configure do |config|
config.mode = :mechanize
end
visit 'http://gmail.com'
assert_contain 'Welcome to Gmail'
I get this error
/usr/lib/ruby/gems/1.8/gems/webrat-0.6.0/lib/webrat/core/matchers/have_content.rb:57:in 'assert_contain': undefined method assert' for #<Object:0xb7e01958> (NoMethodError)
assert_contain and other assertions are methods of test/unit, try to require it and use webrat from inside a test method:
require 'test/unit'
class TC_MyTest < Test::Unit::TestCase
def test_fail
assert(false, 'Assertion was false.')
end
end
anyway i haven't tested it but I have a working spec_helper for rspec if this can interest you:
require File.dirname(__FILE__) + "/../config/environment" unless defined?(RAILS_ROOT)
require 'spec/rails'
require "webrat"
Webrat.configure do |config|
config.mode = :rails
end
module Spec::Rails::Example
class IntegrationExampleGroup < ActionController::IntegrationTest
def initialize(defined_description, options={}, &implementation)
defined_description.instance_eval do
def to_s
self
end
end
super(defined_description)
end
Spec::Example::ExampleGroupFactory.register(:integration, self)
end
end
plus a spec:
# remember to require the spec helper
describe "Your Context" do
it "should GET /url" do
visit "/url"
body.should =~ /some text/
end
end
give it a try I found it very useful (more than cucumber and the other vegetables around) when there is no need to Text specs (features) instead of Code specs, that I like the most.
ps you need the rspec gem and it installs the 'spec' command to execute your specs.

Resources