RuntimeError (Circular dependency detected while autoloading constant Apps multithreading - ruby

Question

I'm receiving this error:
RuntimeError (Circular dependency detected while autoloading constant Apps
when I'm multithreading. Here is my code below. Why is this happening?
The reason I am trying to multithread is because I am writing a HTML scraping app.
The call to Nokogiri::HTML(open()) is a synchronous blocking call that takes 1 second to return, and I have 100,000+ pages to visit, so I am trying to run several threads to overcome this issue. Is there a better way of doing this?
class ToolsController < ApplicationController
def getWebsites
t1=Thread.new{func1()}
t2=Thread.new{func1()}
t3=Thread.new{func1()}
t4=Thread.new{func1()}
t5=Thread.new{func1()}
t6=Thread.new{func1()}
t1.join
t2.join
t3.join
t4.join
t5.join
t6.join
end
def func1
puts Thread.current
apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
while apps.size == 1 do
app = apps[0]
puts app.name
puts app.iTunes
doc = Nokogiri::HTML(open(app.iTunes))
array = doc.css('div.app-links a').map { |link|
url = link['href']
url = Domainatrix.parse(url)
url.domain + "." + url.public_suffix
}
array.uniq!
if (array.size > 0)
app.website = array.join(', ')
puts app.website
else
app.website = "NONE"
end
app.save
apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
end
end
end

Answer 1

"require" isn't thread-safe
Change your methods so that everything that is to be "required" is done so before the threads start.
For example:
def get_websites
# values = Apps.all # try uncommenting this line if a second-try is required
ar = Apps.where("apps.website IS NULL")
t1 = Thread.new{ func1(ar) }
t2 = Thread.new{ func1(ar) }
t1.join
t2.join
end
def func1( ar )
apps = ar.order("RANDOM()").limit(1)
while (apps.size == 1)
puts Thread.current
end
end
But as somebody pointed out, the way you're multithreading within the controller isn't advised.