I'm trying to use pools in a project of mine that uses Celluloid. However, whenever I invoke the pool method on a class which includes Celluloid (thus receiving methods from Celluloid::ClassMethods) I consistently get the error:
NoMethodError: undefined method `services' for Celluloid:Module
router at /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-supervision-0.20.6/lib/celluloid/supervision/deprecate/supervise.rb:54
supervise at /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-supervision-0.20.6/lib/celluloid/supervision/deprecate/supervise.rb:6
pool at /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-pool-0.20.5/lib/celluloid/supervision/container/behavior/pool.rb:13
<top> at celluloid_pool_test.rb:14
Specifically, this part seems to be the problem:
NoMethodError: undefined method `services' for Celluloid:Module
It tells me that the offending line is /Users/my_username/.rvm/gems/jruby-9.0.5.0/gems/celluloid-supervision-0.20.6/lib/celluloid/supervision/deprecate/supervise.rb:54. It turns out that line holds the code for the Celluloid::Supervision.router method:
def router(*_args)
# TODO: Actually route, based on :branch, if present; or else:
Celluloid.services ### this line is what causes the error
end
To make sure that the issue wasn't with my particular project, I grabbed a code sample from this article which utilizes pools and tried to run it:
require 'celluloid'
require 'mathn'
class PrimeWorker
include Celluloid
def prime(number)
if number.prime?
puts number
end
end
end
pool = PrimeWorker.pool
(2..1000).to_a.map do |i|
pool.prime! i
end
sleep 100
It failed with the exact same error as my project:
Finally, I ran a dead simple piece of code in IRB to see if pool is what triggers the error about services:
class Foo
include Celluloid
end
Foo.pool
Sure enough, I got the exact same error. It seems that there is a bug in Celluloid or that I'm not loading a dependency properly. However, I did require 'celluloid/supervision' in my attempts at solving this, to no avail. Am I doing something wrong on my end or is this a bug in Celluloid?
It seems that others have run into this issue before: https://github.com/celluloid/celluloid-pool/issues/10. I guess it has something to do with Celluloid.services being deprecated and not working in newer versions of Celluloid, so using require 'celluloid/current' rather than just require 'celluloid' seems to do the trick.
Related
I have this code:
puts require './item'
puts $"
class Light < Item
#code
end
Item class in item.rb:
require './v3d'
require './ray'
class Item
attr_accessor :pos
def initialize(pos)
#pos = pos
end
def check(pos, dir)
return nil
end
def normal(ray)
return nil
end
end
that when I run my program prints this output:
false
enumerator.so
thread.rb
rational.so
complex.so
/usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/encdb.so
/usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/trans/transdb.so
/usr/lib/ruby/2.3.0/unicode_normalize.rb
/usr/lib/x86_64-linux-gnu/ruby/2.3.0/rbconfig.rb
/usr/lib/ruby/2.3.0/rubygems/compatibility.rb
/usr/lib/ruby/2.3.0/rubygems/defaults.rb
/usr/lib/ruby/2.3.0/rubygems/deprecate.rb
/usr/lib/ruby/2.3.0/rubygems/errors.rb
/usr/lib/ruby/2.3.0/rubygems/version.rb
/usr/lib/ruby/2.3.0/rubygems/requirement.rb
/usr/lib/ruby/2.3.0/rubygems/platform.rb
/usr/lib/ruby/2.3.0/rubygems/basic_specification.rb
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb
/usr/lib/ruby/2.3.0/rubygems/util/list.rb
/usr/lib/x86_64-linux-gnu/ruby/2.3.0/stringio.so
/usr/lib/ruby/2.3.0/rubygems/specification.rb
/usr/lib/ruby/2.3.0/rubygems/exceptions.rb
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_gem.rb
/usr/lib/ruby/2.3.0/monitor.rb
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb
/usr/lib/ruby/2.3.0/rubygems.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/version.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/core_ext/name_error.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/levenshtein.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/jaro_winkler.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/spell_checkable.rb
/usr/lib/ruby/2.3.0/delegate.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/spell_checkers/name_error_checkers.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/spell_checkers/method_name_checker.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/spell_checkers/null_checker.rb
/usr/lib/ruby/vendor_ruby/did_you_mean/formatter.rb
/usr/lib/ruby/vendor_ruby/did_you_mean.rb
/home/<user>/Documents/ruby/ray/write_ppm.rb
/home/<user>/Documents/ruby/ray/v3d.rb
/home/<user>/Documents/ruby/ray/pixel.rb
/home/<user>/Documents/ruby/ray/image.rb
/home/<user>/Documents/ruby/ray/material.rb
then throws:
/home/<user>/Documents/ruby/ray/light.rb:4:in `<top (required)>': uninitialized constant Item (NameError)
When require './item' is called, there is no error AND it returns false. From my understanding of how require works, it seems that the program incorrectly thinks it does not need to load item.rb. Why does this happen and how can I fix it?
Edit: expanded on some code
As a generic answer, not related to op, but because I had a similar issue and was pointed here by search engines.
Basically require 'http' returned false while gem was not loaded.
I figured out that there is a http.rb file inside load path and it is being loaded instead of the standard gem. So double check there are no conflicting file names of ruby files under library load path and the gem name.
I solved the problem by totally rewriting my require statements for every file. What I think the problem was, was this:
item.rb contained require './ray'
ray.rb contained require './light'
light.rb contained require './item' and class Light < Item
While loading item.rb, the interpreter saw it needed to also load ray.rb and therefore light.rb. When it reached the require './item' inside light.rb, it returned false because it was in the process of loading that file. However, since it was not yet finished loading, it did not show up in $". The interpreter then needed access to the definition of the Item class to finish loading light.rb, but because it needed to finish loading light.rb to load item.rb, the interpreter thew a NameError.
I think you may want require_relative instead.
http://ruby-doc.org/core-2.4.0/Kernel.html#method-i-require_relative
vs
http://ruby-doc.org/core-2.4.0/Kernel.html#method-i-require
You are correct in saying that if require 'my_lib' returns false, then 'my_lib' has already been loaded. However this is different from saying that a MyLib class is defined. Does your item.rb define an Item class?
Also, it may be possible that Item is defined somewhere else in the namespace hierarchy. e.g. if your item.rb is in some_gem/item.rb, and you're calling require from some_gem/, it will load successfully, but the name of the class might be SomeGem::Item. In this case you wouldn't be able to access it directly from the root namespace.
Last thing I can think of is that the item.rb file is changing under, or otherwise has some very dynamic pieces that are confusing the interpreter.
I would think that the issue is one of these before thinking that require is somehow messing up.
I am running a padrino application and have started with the included mailers. I want to test that a mail is sent and had previously had no trouble accessing the Mail::TestMailer object to look at the mails delivered during the test.
That is the background about what I am doing but not precisely the question. I want to know how can a module become available to the runtime environment.
I have this test in two versions
first
def test_mailer
Mail::TestMailer.deliveries.clear
get '/owners/test'
e = Mail::TestMailer.deliveries.pop
puts e.to.to_s
end
second
def test_mailer
get '/owners/test'
Mail::TestMailer.deliveries.clear
e = Mail::TestMailer.deliveries.pop
puts e.to.to_s
end
In the second version this test fails with the error message NoMethodError: undefined method to' for nil:NilClass This makes sense to me. I clear the messages then ask for the last one which should be nil. However when I run the test on the first version the error is NameError: uninitialized constant OwnersControllerTest::Mail
So somehow the get method is causing the Mail object/module to be made available. I don't understand how it can do this. I don't know if this is a rack-test or padrino thing so am unsure what extra information to copy in here.
Add require 'mail' to your test helper.
The issue is explained here: https://github.com/padrino/padrino-framework/issues/1797
I'm using "therubyracer" in a model, and I'm requiring at the top of the model as so:
require 'v8'
class Thing
def self.ctx; ##ctx ||= V8::Context.new; end;
def self.eval(script); ctx.eval(script); end;
end
However, I intermittently get:
NameError - uninitialized constant Thing::V8:
/app/thing.rb:3:in `ctx'
When testing requests through a local Padrino server, apparently after I modify code in Thing. This is corrected by restarting the padrino server. I'm assuming requiring v8 somewhere else would fix this problem, wheres the correct place?
This looks like it might be caused by the Padrino reloader getting confused when it reloads your thing.rb file, causing Ruby to look for V8 in the Thing namespace.
Try explicitly specifying V8 is in the top level using the :: prefix:
def self.ctx; ##ctx ||= ::V8::Context.new; end;
You can put it wherever you want if you add it on the Gemfile. Did you added it?
Thanks!
I just switched to using Sidekiq on Heroku but I'm getting the following after my jobs run for a while:
2012-12-11T09:53:07+00:00 heroku[worker.1]: Process running mem=1037M(202.6%)
2012-12-11T09:53:07+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-12-11T09:53:28+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-12-11T09:53:28+00:00 heroku[worker.1]: Process running mem=1044M(203.9%)
It keeps growing like that.
For these jobs I'm using Nokogiri and HTTParty to retrieve URLs and parse them. I've tried changing some code but I'm not actually sure what I'm looking for in the first place. How should I go about debugging this?
I tried adding New Relic to my app but unfortunately that doesn't support Sidekiq yet.
Also, after Googling I'm trying to switch to a SAX parser and see if that works but I'm getting stuck. This is what I've done so far:
class LinkParser < Nokogiri::XML::SAX::Document
def start_element(name, attrs = [])
if name == 'a'
puts Hash[attrs]['href']
end
end
end
Then I try something like:
page = HTTParty.get("http://site.com")
parser = Nokogiri::XML::SAX::Parser.new(LinkParser.new)
Then I tried using the following methods with the data I retrieved using HTTParty, but haven't been able to get any of these methods to work correctly:
parser.parse(File.read(ARGV[0], 'rb'))
parser.parse_file(filename, encoding = 'UTF-8')
parser.parse_memory(data, encoding = 'UTF-8')
Update
I discovered that the parser wasn't working because I was calling parser.parse(page) instead of parser.parse(page.body) however I've tried printing out all the html tags for various websites using the above script and for some sites it prints out all the tags, while for others it only prints out a few tags.
If I use Nokogiri::HTML() instead of parser.parse() it works fine.
I was using Nokogiri::XML::SAX::Parser.new() instead of Nokogiri::HTML::SAX::Parser.new() for HTML documents and that's why I was running into trouble.
Code Update
Ok, I've got the following code working now, but can't figure out how to put the data I get into an array which I can use later on...
require 'nokogiri'
class LinkParser < Nokogiri::XML::SAX::Document
attr_accessor :link
def initialize
#link = false
end
def start_element(name, attrs = [])
url = Hash[attrs]
if name == 'a' && url['href'] && url['href'].starts_with?("http")
#link = true
puts url['href']
puts url['rel']
end
end
def characters(anchor)
puts anchor if #link
end
def end_element(name)
#link = false
end
def self.starts_with?(prefix)
prefix.respond_to?(:to_str) && self[0, prefix.length] == prefix
end
end
In the end I discovered that the memory leak is due to the 'Typhoeus' gem which is a dependency for the 'PageRankr' gem that I'm using in part of my code.
I discovered this by running the code locally while monitoring memory usage with watch "ps u -C ruby", and then testing different parts of the code until I could pinpoint where the memory leak came from.
I'm marking this as the accepted answer since in the original question I didn't know how to debug memory leaks but someone told me to do the above and it worked.
Just in case if you can't to resolve gems memory leaks issue:
You can run sidekiq jobs inside a forks, as described in the answer https://stackoverflow.com/a/1076445/3675705
Just add Application helper "do_in_child" and then inside your worker
def perform
do_in_child do
# some polluted task
end
end
Yes, i know it's kind a dirty solution becase Sidekiq should work in threads, but in my case it's the only one fast solution for production becase i have a slow jobs with parsing big XML files by nokogiri.
"Fast" thread feature will not give any advantage but memory leaks gives me a 2GB+ main sidekiq process after 10 minutes of work. And after one day sidekiq virtual memory grows up to 11GB (all available virtual memory on my server) and all the tasks are going extremely slow.
Essentially, I want to create a program that will run some untrusted code that defines some method or class, and then run an untrusted rspec spec against it.
I've looked into sandboxing Ruby a bit, and this video from rubyconf was particularly helpful. After looking at several solutions, the two that appear to be the most helpful are rubycop, which essentially does static analysis on the code, and the jruby sandbox (both covered in above video). My instinct tells me that the jruby sandbox is probably safer, but I could well be wrong.
Here's a completely unsafe example of what I want to do:
code = <<-RUBY
class Person
def hey
"hey!"
end
end
RUBY
spec = <<-RUBY
describe Person do
let(:person) { Person.new }
it "says hey" do
person.hey.should == "hey!"
end
end
RUBY
# code and spec will be from user input (unsafe)
eval code
require 'rspec/autorun'
eval spec
Which all works fine, but the code obviously needs to be sandboxed. It will be a matter of minutes before some genius submits system("rm -rf /*"), fork while fork or something equally dangerous.
I made various attempts with the jruby sandbox...
sand = Sandbox::Safe.new
sand.eval("require 'rspec/autorun'")
sand.activate! # lock it down
sand.eval code
puts sand.eval spec
That code throws this exception:
Sandbox::SandboxException: NoMethodError: undefined method `require' for #<RSpec::Core::Configuration:0x7c3cfaab>
This is because RSpec tries to require some stuff after the sandbox has been locked down.
So, I tried to force RSpec to require stuff before the sandbox gets locked down by calling an empty describe:
sand = Sandbox::Safe.new
sand.eval("require 'rspec/autorun'")
sand.eval("describe("") { }")
sand.activate! # lock it down
sand.eval code
sand.eval spec
And I get this:
Sandbox::SandboxException: NameError: uninitialized constant RSpec
Which basically means that RSpec doesn't exist in the sandbox. Which is odd, considering sand.eval("require 'rspec/autorun'") returns true, and that the earlier example actually worked (RSpec's autoloader started to run).
It may be a problem with gems and this particular sandbox though. The sandbox object actually supports a method #require, which is essentially bound to Kernel.require, and therefore can't load gems.
It's starting to look like using this sandbox just might not really be possible with rspec. The main problem is trying to actually load it into the sandbox. I even tried something like this:
require 'rspec'
sand.ref(RSpec) # open access to local rspec
But it wasn't having any of it.
So, my question is two-fold:
Does anyone have any bright ideas on how to get this to work with the jruby sandbox?
If not, how secure is rubycop? Apparently codeschool use it, so it must be pretty well tested... it would be nice to be able to use ruby 1.9 instead of jruby as well.
It looks like the sand box environment isn't loading the bundle/gemset. RVM could be at fault here if you are using a gemset or something.
One might try loading the Bundle again once sand boxed.
I would look at ruby taint modes
$SAFE The security level
0 --> No checks are performed on externally supplied (tainted) data. (default)
1 --> Potentially dangerous operations using tainted data are forbidden.
2 --> Potentially dangerous operations on processes and files are forbidden.
3 --> All newly created objects are considered tainted.
4 --> Modification of global data is forbidden.
I have been trying to figure out a similar problem. I want to use some gems like json and rest-client inside my sandbox after activating it. I tried following.
require "sandbox"
s=Sandbox.safe
s.eval <<-RUBY
require 'bundler'
Bundler.require :sandbox
RUBY
s.activate!
Gemfile.rb
group :sandbox do
platforms :jruby do
gem 'json'
gem 'rest-client'
end
end
This way, I was able to require gems in my sandbox. But, then there were some gem specific issues with sandbox. For eg, I had to add a method initialize_dup to whitelist for safe.rb in jruby-sandbox. RestClient has some problem with Fake File Sytem ALT_SEPARATOR which I am trying to patch. You can try this approach for RSpec and see if everything goes through.