How can I test that the number of successfully returned threads to the main thread is equal to the number of threads created in rspec?
describe '#update_tokens' do
it 'is threadsafe', db_strategy: :truncation do
u = create(:user)
thread_injected = false
time = Time.now
allow(Time).to receive(:now) do
unless thread_injected
thread_injected = true
Thread.new { u.reload.update_tokens('c2', 't2') }.join
end
time
end
u.update_tokens('c1', 't1')
expect(u.tokens['c1']).not_to be_nil
expect(u.tokens['c2']).not_to be_nil
end
end
Original: http://blog.paulrugelhiatt.com/rails/rspec/2015/09/28/testing-thread-safety-concurrency-in-rspec.html
Related
I have been experimenting multi-threading concept in Ruby for the past a week.
For practising, I am designing a file downloader that makes parallel requests for a collection of URLs. Currently I need to safely shutdown threads when interrupt signal is triggered. I have read the theory of multi-threading and catching a signal at runtime. Yet despite the whole those theoretical knowledge, I still don't have any idea about how to use them in practice.
I am leaving my proof of concept work below, anyhow.
class MultiThread
attr_reader :limit, :threads, :queue
def initialize(limit)
#limit = limit
#threads = []
#queue = Queue.new
end
def add(*args, &block)
queue << [block, args]
end
def invoke
1.upto(limit).each { threads << spawn_thread }
threads.each(&:join)
end
private
def spawn_thread
Thread.new do
Thread.handle_interrupt(RuntimeError => :on_blocking) do
# Nothing to do
end
until queue.empty?
block, args = queue.pop
block&.call(*args)
end
end
end
end
urls = %w[https://example.com]
thread = MultiThread.new(2)
urls.each do |url|
thread.add do
puts "Downloading #{url}..."
sleep 1
end
end
thread.invoke
Yeah, the docs for handle_interrupt are confusing. Try this, which I based on the connection_pool gem used by e.g. puma.
$stdout.sync = true
threads = 3.times.map { |i|
Thread.new {
Thread.handle_interrupt(Exception => :never) do
begin
Thread.handle_interrupt(Exception => :immediate) do
puts "Thread #{i} doing work"
sleep 1000
end
ensure
puts "Thread #{i} cleaning up"
end
end
}
}
Signal.trap("INT") {
puts 'Exiting gracefully'
threads.each { |t|
puts 'killing thread'
t.kill
}
exit
}
threads.each { |t| t.join }
Output:
Thread 1 doing work
Thread 2 doing work
Thread 0 doing work
^CExiting gracefully
killing thread
killing thread
killing thread
Thread 0 cleaning up
Thread 1 cleaning up
Thread 2 cleaning up
I am having difficulty writing integration (no stubbing) tests for the following scenario: a process (rake task) that runs in a loop, emitting some values. Below is an approximation of the use case.
The test will succeed if I control-C it, but I would like it to catch the success condition and stop.
Anyone has some good suggestions? (stubbing/mocking are not good suggestions). I guess may be there is a way to instruct RSpec to stop a process after a matcher returns success?
describe 'rake reactor' do
it 'eventually returns 0.3' do
expect { Rake::Task['reactor'].execute }.to output(/^0\.3.*/).to_stdout
end
end
class Reactor
def initialize
#stop = false
end
def call
loop do
break if stop?
sleep random_interval
yield random_interval
end
end
def stop
#stop = true
end
def stop?
#stop == true
end
def random_interval
rand(0.1..0.4)
end
end
desc 'Start reactor'
task reactor: :environment do
reactor = Reactor.new
trap(:INT) do
reactor.stop
end
reactor.call { |m| p m }
end
A naïve way to handle it is to start a new thread and send INT from there after some predefined timeout:
before do
Thread.new do
sleep 0.5
Process.kill('INT', Process.pid)
end
end
I currently have a script written in Ruby that scans a range of IP addresses and tries to connect to them. It's extremely slow at the moment. It takes up to 300 seconds to scan 254 hosts on the network, and that's obviously not very practical. What I'm trying to do is give the script some concurrency in hopes of speeding up the script. So far this is what I have:
require 'socket'
require 'celluloid'
$res_arr = []
class Ranger
include Celluloid
def initialize(host)
#host = host
#timeout = 1
end
def ip_range(host)
host =~ /(?:\d{1,3}\.){3}[xX*]{1,3}/
end
def ctrl(host)
begin
if ip_range(host)
strIP = host.gsub(/[xX*]/, '')
(1..254).each do |oct|
$res_arr << strIP+oct.to_s
end
else
puts "Invalid host!"
end
rescue
puts "onnection terminated."
end
end
def connect
addr = Socket.getaddrinfo(#host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
$res_arr << "#{#host} Firewalled!"
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
$res_arr << "#{#host}Connected!"
end
rescue Errno::ECONNREFUSED
$res_arr << "#{#host} Refused!"
rescue
false
end
end
sock
end
def output(contents)
puts contents.value
end
end # Ranger
main = Ranger.new(ARGV[0])
main.ctrl(ARGV[0])
$res_arr.each do |ip|
scan = Ranger.new(ip)
scnftr = scan.future :connect
scan.output(scnftr)
end
The script works, but it takes just as long as before I included Celluloid at all. Am I misunderstanding how Celluloid works and what it's supposed to do?
Your problem is that each iteration of your loop starts a future, then immediately waits for it to return a value. What you want instead is start all futures, then wait for all futures to finish in two separate steps:
futures = $res_arr.map do |ip|
scan = Ranger.new(ip)
scan.future :connect
end
# now that all futures are running, we can start
# waiting for the first one to finish
futures.each do |future|
puts future.value
end
Here's another example from the celluloid source: https://github.com/celluloid/celluloid/blob/master/examples/simple_pmap.rb
Starting using rspec I have difficulties trying to test threaded code.
Here is a simplicfication of a code founded, and I made it cause i need a Queue with Timeout capabilities
require "thread"
class TimeoutQueue
def initialize
#lock = Mutex.new
#items = []
#new_item = ConditionVariable.new
end
def push(obj)
#lock.synchronize do
#items.push(obj)
#new_item.signal
end
end
def pop(timeout = :never)
timeout += Time.now unless timeout == :never
#lock.synchronize do
loop do
time_left = timeout == :never ? nil : timeout - Time.now
if #items.empty? and time_left.to_f >= 0
#new_item.wait(#lock, time_left)
end
return #items.shift unless #items.empty?
next if timeout == :never or timeout > Time.now
return nil
end
end
end
alias_method :<<, :push
end
But I can't find a way to test it using rspec. Is there any effective documentation on testing threaded code? Any gem that can helps me?
I'm a bit blocked, thanks in advance
When unit-testing we don't want any non-deterministic behavior to affect our tests, so when testing threading we should not run anything in parallel.
Instead, we should isolate our code, and simulate the cases we want to test, by stubbing #lock, #new_item, and perhaps even Time.now (to be more readable I've taken the liberty to imagine you also have attr_reader :lock, :new_item):
it 'should signal after push' do
allow(subject.lock).to receive(:synchronize).and_yield
expect(subject.new_item).to receive(:signal)
subject.push('object')
expect(subject.items).to include('object')
end
it 'should time out if taken to long to enter synchronize loop' do
#now = Time.now
allow(Time).to receive(:now).and_return(#now, #now + 10.seconds)
allow(subject.items).to receive(:empty?).and_return true
allow(subject.lock).to receive(:synchronize).and_yield
expect(subject.new_item).to_not receive(:wait)
expect(subject.pop(5.seconds)).to be_nil
end
etc...
I'm reading a Redis set within an EventMachine reactor loop using a suitable Redis EM gem ('em-hiredis' in my case) and have to check if some Redis sets contain members in a cascade. My aim is to get the name of the set which is not empty:
require 'eventmachine'
require 'em-hiredis'
def fetch_queue
#redis.scard('todo').callback do |scard_todo|
if scard_todo.zero?
#redis.scard('failed_1').callback do |scard_failed_1|
if scard_failed_1.zero?
#redis.scard('failed_2').callback do |scard_failed_2|
if scard_failed_2.zero?
#redis.scard('failed_3').callback do |scard_failed_3|
if scard_failed_3.zero?
EM.stop
else
queue = 'failed_3'
end
end
else
queue = 'failed_2'
end
end
else
queue = 'failed_1'
end
end
else
queue = 'todo'
end
end
end
EM.run do
#redis = EM::Hiredis.connect "redis://#{HOST}:#{PORT}"
# How to get the value of fetch_queue?
foo = fetch_queue
puts foo
end
My question is: how can I tell EM to return the value of 'queue' in 'fetch_queue' to use it in the reactor loop? a simple "return queue = 'todo'", "return queue = 'failed_1'" etc. in fetch_queue results in "unexpected return (LocalJumpError)" error message.
Please for the love of debugging use some more methods, you wouldn't factor other code like this, would you?
Anyway, this is essentially what you probably want to do, so you can both factor and test your code:
require 'eventmachine'
require 'em-hiredis'
# This is a simple class that represents an extremely simple, linear state
# machine. It just walks the "from" parameter one by one, until it finds a
# non-empty set by that name. When a non-empty set is found, the given callback
# is called with the name of the set.
class Finder
def initialize(redis, from, &callback)
#redis = redis
#from = from.dup
#callback = callback
end
def do_next
# If the from list is empty, we terminate, as we have no more steps
unless #current = #from.shift
EM.stop # or callback.call :error, whatever
end
#redis.scard(#current).callback do |scard|
if scard.zero?
do_next
else
#callback.call #current
end
end
end
alias go do_next
end
EM.run do
#redis = EM::Hiredis.connect "redis://#{HOST}:#{PORT}"
finder = Finder.new(redis, %w[todo failed_1 failed_2 failed_3]) do |name|
puts "Found non-empty set: #{name}"
end
finder.go
end