I'm trying to get Stalker working with Beanstalkd in my Ramaze application. I can enqueue jobs no problem, but my worker doesn't recognize my custom models. Here is my jobs.rb:
require 'stalker'
include Stalker
BEANSTALK_URL = '0.0.0.0:11300'
job 'award_badges' do |args|
Badge.award_badges(args[:user_id], args[:badge_category])
end
I run the jobs.rb with stalker jobs.rb and always receive the error: Exception NameError -> uninitialized constant Badge
In all the stalker examples no libraries are included, how does Stalker know about those classes?
This is because you're not loading your enviroment with your jobs, here is a well formatted jobs.rb:
require File.expand_path("../environment", __FILE__)
require 'stalker'
include Stalker
job "my_job" do |args|
# Do something here...
end
As you can see in the first three lines we first require our rails enviroment and then stalker, so it can be aware about your models and stuff.
Cheers!
Related
I'm trying to use pools in a project of mine that uses Celluloid. However, whenever I invoke the pool method on a class which includes Celluloid (thus receiving methods from Celluloid::ClassMethods) I consistently get the error:
NoMethodError: undefined method `services' for Celluloid:Module
router at /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-supervision-0.20.6/lib/celluloid/supervision/deprecate/supervise.rb:54
supervise at /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-supervision-0.20.6/lib/celluloid/supervision/deprecate/supervise.rb:6
pool at /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-pool-0.20.5/lib/celluloid/supervision/container/behavior/pool.rb:13
<top> at celluloid_pool_test.rb:14
Specifically, this part seems to be the problem:
NoMethodError: undefined method `services' for Celluloid:Module
It tells me that the offending line is /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-supervision-0.20.6/lib/celluloid/supervision/deprecate/supervise.rb:54. It turns out that line holds the code for the Celluloid::Supervision.router method:
def router(*_args)
# TODO: Actually route, based on :branch, if present; or else:
Celluloid.services ### this line is what causes the error
end
To make sure that the issue wasn't with my particular project, I grabbed a code sample from this article which utilizes pools and tried to run it:
require 'celluloid'
require 'mathn'
class PrimeWorker
include Celluloid
def prime(number)
if number.prime?
puts number
end
end
end
pool = PrimeWorker.pool
(2..1000).to_a.map do |i|
pool.prime! i
end
sleep 100
It failed with the exact same error as my project:
Finally, I ran a dead simple piece of code in IRB to see if pool is what triggers the error about services:
class Foo
include Celluloid
end
Foo.pool
Sure enough, I got the exact same error. It seems that there is a bug in Celluloid or that I'm not loading a dependency properly. However, I did require 'celluloid/supervision' in my attempts at solving this, to no avail. Am I doing something wrong on my end or is this a bug in Celluloid?
It seems that others have run into this issue before: https://github.com/celluloid/celluloid-pool/issues/10. I guess it has something to do with Celluloid.services being deprecated and not working in newer versions of Celluloid, so using require 'celluloid/current' rather than just require 'celluloid' seems to do the trick.
I'm using "therubyracer" in a model, and I'm requiring at the top of the model as so:
require 'v8'
class Thing
def self.ctx; ##ctx ||= V8::Context.new; end;
def self.eval(script); ctx.eval(script); end;
end
However, I intermittently get:
NameError - uninitialized constant Thing::V8:
/app/thing.rb:3:in `ctx'
When testing requests through a local Padrino server, apparently after I modify code in Thing. This is corrected by restarting the padrino server. I'm assuming requiring v8 somewhere else would fix this problem, wheres the correct place?
This looks like it might be caused by the Padrino reloader getting confused when it reloads your thing.rb file, causing Ruby to look for V8 in the Thing namespace.
Try explicitly specifying V8 is in the top level using the :: prefix:
def self.ctx; ##ctx ||= ::V8::Context.new; end;
You can put it wherever you want if you add it on the Gemfile. Did you added it?
Thanks!
I just switched to using Sidekiq on Heroku but I'm getting the following after my jobs run for a while:
2012-12-11T09:53:07+00:00 heroku[worker.1]: Process running mem=1037M(202.6%)
2012-12-11T09:53:07+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-12-11T09:53:28+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-12-11T09:53:28+00:00 heroku[worker.1]: Process running mem=1044M(203.9%)
It keeps growing like that.
For these jobs I'm using Nokogiri and HTTParty to retrieve URLs and parse them. I've tried changing some code but I'm not actually sure what I'm looking for in the first place. How should I go about debugging this?
I tried adding New Relic to my app but unfortunately that doesn't support Sidekiq yet.
Also, after Googling I'm trying to switch to a SAX parser and see if that works but I'm getting stuck. This is what I've done so far:
class LinkParser < Nokogiri::XML::SAX::Document
def start_element(name, attrs = [])
if name == 'a'
puts Hash[attrs]['href']
end
end
end
Then I try something like:
page = HTTParty.get("http://site.com")
parser = Nokogiri::XML::SAX::Parser.new(LinkParser.new)
Then I tried using the following methods with the data I retrieved using HTTParty, but haven't been able to get any of these methods to work correctly:
parser.parse(File.read(ARGV[0], 'rb'))
parser.parse_file(filename, encoding = 'UTF-8')
parser.parse_memory(data, encoding = 'UTF-8')
Update
I discovered that the parser wasn't working because I was calling parser.parse(page) instead of parser.parse(page.body) however I've tried printing out all the html tags for various websites using the above script and for some sites it prints out all the tags, while for others it only prints out a few tags.
If I use Nokogiri::HTML() instead of parser.parse() it works fine.
I was using Nokogiri::XML::SAX::Parser.new() instead of Nokogiri::HTML::SAX::Parser.new() for HTML documents and that's why I was running into trouble.
Code Update
Ok, I've got the following code working now, but can't figure out how to put the data I get into an array which I can use later on...
require 'nokogiri'
class LinkParser < Nokogiri::XML::SAX::Document
attr_accessor :link
def initialize
#link = false
end
def start_element(name, attrs = [])
url = Hash[attrs]
if name == 'a' && url['href'] && url['href'].starts_with?("http")
#link = true
puts url['href']
puts url['rel']
end
end
def characters(anchor)
puts anchor if #link
end
def end_element(name)
#link = false
end
def self.starts_with?(prefix)
prefix.respond_to?(:to_str) && self[0, prefix.length] == prefix
end
end
In the end I discovered that the memory leak is due to the 'Typhoeus' gem which is a dependency for the 'PageRankr' gem that I'm using in part of my code.
I discovered this by running the code locally while monitoring memory usage with watch "ps u -C ruby", and then testing different parts of the code until I could pinpoint where the memory leak came from.
I'm marking this as the accepted answer since in the original question I didn't know how to debug memory leaks but someone told me to do the above and it worked.
Just in case if you can't to resolve gems memory leaks issue:
You can run sidekiq jobs inside a forks, as described in the answer https://stackoverflow.com/a/1076445/3675705
Just add Application helper "do_in_child" and then inside your worker
def perform
do_in_child do
# some polluted task
end
end
Yes, i know it's kind a dirty solution becase Sidekiq should work in threads, but in my case it's the only one fast solution for production becase i have a slow jobs with parsing big XML files by nokogiri.
"Fast" thread feature will not give any advantage but memory leaks gives me a 2GB+ main sidekiq process after 10 minutes of work. And after one day sidekiq virtual memory grows up to 11GB (all available virtual memory on my server) and all the tasks are going extremely slow.
I am getting this error While running this
LoadError: Expected /home/user/Desktop/Tripurari/myapp/app/models/host.rb to define Host##
But every thing on it's place. Can some one tell me what the exact problem is below method.
def self.check_all(keyword)
memo_mutex = Mutex.new
memo = {}
threads = []
name = keyword.keyword
SITES.each do |site_and_options|
threads << Thread.new do
#host = Host.find_or_create_by_name(site)
if keyword.unavailable_usernames.find_by_host_id(#host.id)
memo[#host.name] = true
else
memo[#host.name] = false
end
end
end
threads.each { |t| t.join }
memo
end
The issue is probably caused by the autoloader. If the Host class is not yet loaded when first entering the loop where you create a couple of new threads, it is autoloaded, i.e. Rails searches the loadpath for a file matching the naming conventions and requires it.
This process is not threadsave. In your case, as you are creating servral threads in quick succession, each trying to autoload the global class, you get race conditions and strange things happen. Basically, you have two options for tackling this:
You can explicitly load the model before starting your threads by using require 'host' before starting your loop.
Or you can set config.threadsave! in an initializer. This will (among other things) preload all your classes when starting your server. This is preferred as with this, you avoid a truckload of other difficult to debug concurrency issues. For more information about config.threadsafe!, please refer to the excellent article by Aaron Patterson arguing it should be removed altogether in Rails 4.
Assuming the code you've quoted above is in a model's .rb file, add require_relative "host" to the top of that file.
I've seen this asked a few times in other threads, but none of the answers seem to apply.
Environment:
Rails 3
amazon/ecs gem from jugend. The lone file is here:
http://github.com/jugend/amazon-ecs/blob/master/lib/amazon/ecs.rb
my gemfile has:
gem 'amazon-ecs', :git => 'git://github.com/jugend/amazon-ecs.git'
Everything works in irb. I can run:
bundle console
require 'amazon/ecs' and then go to town
when I try to use it from the controller though, like so:
require 'amazon/ecs'
require 'amazon/ecs'
class SearchController < ApplicationController
def index
end
def results
Amazon::Ecs.configure do |options|
options[:aWS_access_key_id] = '[key]'
options[:aWS_secret_key] = '[secret]'
end
res = Amazon::Ecs.item_search(params[:search], {:response_group => 'Medium', :search_index => 'All'})
end
end
I get: uninitialized constant SearchController::Amazon at line 8, where I first try to use Amazon.
the ecs.rb has a module Amazon containing a class Ecs. I'm not sure why this is working in erb, and not in rails.
I'm still kinda new to Rails, so please answer using small words. :-/
Was given the answer. I moved my initialization code to an initializer in config/initializers file, removed the require entirely, and things worked. I don't know why though, so if someone could answer that, that'd be great.
All of the gems require their files by default, so usually you don't need to explicitly require any files.
Speaking about your problem, it could somehow be, that your controller is run before Amazon module is processed.