Limiting the number of threads to be executed - ruby

Question

I am trying to limit the number of threads that must be executed simultaneously or to separate the elements of the array in a certain amount and send them separately (splice) but I have no idea where and how to start. The code is executing the Threads quickly and this is hindering many.
My code:
file = File.open('lista.txt').read
file.gsub!(/\r\n?/, "")
arr = []
file.each_line do |log|
arr << Thread.new{
puts login = Utils.req("GET","http://httpbin.org/GET",{
'Host': 'api.httpbin.org',
'content-type': 'application/json',
'accept': '*/*',
},
false)
}
end
arr.each(&:join)

Answer 1

If you want to limit how many threads are running concurrently, use a thread pool. This creates a "pool" of worker threads and limits the number of threads running at the same time. You can write your own, or use a gem like celluloid.
class Whatever
include Celluloid
def login_request
Utils.req(
"POST",
"http://httpbin.org/post",
{
'Host': 'api.httpbin.org',
'content-type': 'application/json',
'accept': '*/*',
},
false
)
end
end
# Only 5 threads will run at a time.
pool = Whatever.pool(size: 5)
file.each_line do |log|
split = log.split("|")
id = split[0]
number = split[1]
# If there's a free working in the pool, this will execute immediately.
# If not, it will wait until there is a free worker.
pool.login_request
end

Answer 2

You could use a thread pool. Here's a very basic one using Queue:
queue = Queue.new
pool = Array.new(5) do
Thread.new do
loop do
line = queue.pop
break if line == :stop
# do something with line, e.g.
# id, number = line.split('|')
# Utils.req(...)
end
end
end
It creates 5 threads and stores them in an array for later reference. Each thread runs a loop, calling Queue#pop to fetch a line from the queue. If the queue is empty, pop will suspend the thread until data becomes available. So initially, the threads will just sit there waiting for work to come:
queue.num_waiting #=> 5
pool.map(&:status) #=> ["sleep", "sleep", "sleep", "sleep", "sleep"]
Once the thread has retrieved a line, it will process it (to be implemented) and fetch a new one (or fall asleep again if there is none). If the line happens to be the symbol :stop, the thread will break the loop and terminate. (a very pragmatic approach)
To fill the queue, we can use a simple loop in our main thread:
file.each_line { |line| queue << line }
Afterwards, we push 5 :stop symbols to the end of the queue and then wait for the threads to finish:
pool.each { queue << :stop }
pool.each(&:join)
At this point, no threads are waiting for the queue – they all terminated normally:
queue.num_waiting #=> 0
pool.map(&:status) #=> [false, false, false, false, false]
Note that Queue isn't limited to strings or symbols. You can push any Ruby object to it. So instead of pushing the raw line, you could push the pre-processed data:
file.each_line do |line|
id, number = line.split('|')
queue << [id, number]
end
This way, your threads don't have to know about the log format. They can just work on id and number:
loop do
value = queue.pop
break if value == :stop
id, number = value
# do something with id and number
end
Instead of writing your own thread pool, you could of course use one of the various concurrency gems.