Is it possible to create a "worker thread" so to speak that is on standby until it receives a function to execute asynchronously?
Is there a way to send a function like
def some_function
puts "hi"
# write something
db.exec()
end
to an existing thread that's just sitting there waiting?
The idea is I'd like to pawn off some database writes to a thread which runs asynchronously.
I thought about creating a Queue instance, then have a thread do something like this:
$command = Queue.new
Thread.new do
while trigger = $command.pop
some_method
end
end
$command.push("go!")
However this does not seem like a particularly good way to go about it. What is a better alternative?
The thread gem looks like it would suit your needs:
require 'thread/channel'
def some_method
puts "hi"
end
channel = Thread.channel
Thread.new do
while data = channel.receive
some_method
end
end
channel.send("go!")
channel.send("ruby!") # Any truthy message will do
channel.send(nil) # Non-truthy message to terminate other thread
sleep(1) # Give other thread time to do I/O
The channel uses ConditionVariable, which you could use yourself if you prefer.
Related
In RSpec, I have function that creates a new thread, and inside that thread performs some action–in my case, calls TCPSocket#readline. Here's the function as it is right now:
def read
Thread.new do
while line = #socket.readline
#TODO: stuff
end
end
end
Due to thread scheduling, my test will fail if written as such:
it "reads from socket" do
subject.socket.should_receive(:readline)
subject.read
end
Currently the only way I know to hack around this is to use sleep 0.1. Is there a way to properly delay the test until that thread is running?
If your goal is to assert the system state is changed by the execution of your second thread, you should join on the second thread in your main test thread:
it "reads from socket" do
subject.socket.should_receive(:readline)
socket_thread = subject.read
socket_thread.join
end
This is a bit of a hack, but here's a before block you can use in case you'd like the thread to yield but be able to call join at the end of the thread.
before do
allow(Thread).to receive(:new).and_yield.and_return(Class.new { def join; end }.new)
end
The idea is that the input will be provided by the console, and will be identified with a unique 'id' as the first word of the input. A new thread is spawned when a new id is encountered, with the subsequent input being 'start'. The thread should close when an input with the same id will say 'close'. The ordering of the statements is random.
The trick here is to have a single thread (most easily the main thread) do all the reading. That thread will parse the command, start or stop threads, and dispatch commands to the threads. Each thread will have its own Queue from which it reads commands and into which the main thread puts the commands. Let's see how it might be done.
We'll start with a little module for the control commands, so that they're DRY:
module ControlCommands
START_COMMAND = 'start'
STOP_COMMAND = 'stop'
end
Now let's see the "main":
class Main
def initialize
#workers = Workers.new
#console = Console.new(#workers)
end
def run
#console.read_and_dispatch
#workers.join
end
end
Main.new.run
There's nothing much going on here. We make a console, and tell it to read commands and dispatch them to the workers. The console does not return from that until it has run out of input. The call to #workers.join make sure all the workers have finished working and properly shut down before the program exits.
Here's the Console class:
class Console
def initialize(workers)
#workers = workers
end
def read_and_dispatch
while input = gets
#workers.dispatch *parse_input(input)
end
end
private
def parse_input(input)
input.match(/^(\w+) *(.*)$/)[1..2]
end
end
read_and_dispatch is the main loop. All that it is responsible for is reading and parsing input. As long as there is input, it splits it into the worker name and command, and then tells the workers to handle the command.
Here's the Workers class:
class Workers
include ControlCommands
def initialize
#workers = {}
end
def dispatch(worker_name, command)
case command
when START_COMMAND
start_worker worker_name
when STOP_COMMAND
stop_worker worker_name
else
dispatch_worker worker_name, command
end
end
def join
#workers.each do |worker_name, worker|
worker.stop
worker.join
end
end
private
def start_worker(worker_name)
#workers[worker_name] = Worker.new(worker_name)
end
def stop_worker(worker_name)
#workers[worker_name].stop
end
def dispatch_worker(worker_name, command)
#workers[worker_name].dispatch command
end
end
This is where most of the meat is. This class creates workers (threads), stop them, and dispatches commands to them. Note that there is no error handling here: If you try to stop a thread that isn't started, start one that's already started, or dispatch a command to one that doesn't exist, the program will crash or misbehave. I'll leave the handling of those situations as an "exercise for the reader."
Here's the Worker class:
class Worker
include ControlCommands
def initialize(name)
#name = name
#commands = Queue.new
#thread = start_thread
end
def dispatch(command)
#commands.enq command
end
def stop
#commands.enq STOP_COMMAND
end
def join
#thread.join
end
private
def start_thread
Thread.new do
loop do
command = #commands.deq
break if command == STOP_COMMAND
process_command command
end
end
end
def process_command(command)
print "thread #{#name} received command #{command.inspect}\n"
end
end
This class contains the thread, and the queue used to communicate the between the main thread (the one reading the console) and the worker thread. That queue is also used to stop a thread, synchronously, by putting a STOP_COMMAND in the queue which the thread responds to by exiting. It is best, when you can afford to, to stop threads synchronously rather than asynchronously.
Here's a simple input file:
foo start
bar start
foo input
bar input
foo stop
bar stop
And the output when the program is presented with that input:
thread bar received command "input"
thread foo received command "input"
Is it OK not to call Thread#join? In this case, I don't care if the thread blows up - I just want Unicorn to keep processing.
class MyMiddleware
def initialize(app)
#app = app
end
def call(env)
t = Thread.new { sleep 1 }
t.join # is it ok if I skip this?
#app.call env
end
end
Will I get "zombie threads" or something like that?
It's perfectly fine to not call join - in fact, join is often not needed at all with multithreaded code. You should only call join if you need to block until the new thread completes.
You won't get a "zombie" thread. The new thread will run until completion, then clean itself up for you.
I'm trying to do something like this, where I have two loops going in seperate threads. The problem I am having is that in the main thread, when I use gets and the script is waiting for user input, the other thread is stopped to wait as well.
class Server
def initialize
#server = TCPServer.new(8080)
run
end
def run
#thread = Thread.new(#server) { |server|
while true
newsock = server.accept
puts "some stuff after accept!"
next if !newsock
# some other stuff
end
}
end
end
def processCommand
# some user commands here
end
test = Server.new
while true do
processCommand(STDIN.gets)
end
The above is just a sample of what I want to do.
Is there a way to make the main thread block while waiting for user input?
You might want to take a look at using the select method of the IO class. Take a look at
good select example for handling select with asynchronous input. Depending upon what version of ruby you're using you might have issues with STDIN though, I'm pretty sure it always triggers the select in 1.8.6.
I'm not sure if this is what you are looking for, but I was looking for something similar and this example does exactly what I wanted. The thread will continue processing until the user hits enter, and then the thread will be able to handle your user input as desired.
user_input = nil
t1 = Thread.new do
while !user_input
puts "Running"
end
puts "Stopping per user input: #{user_input}"
end
user_input = STDIN.gets
t1.join
I'm writing a delayed_job clone for DataMapper. I've got what I think is working and tested code except for the thread in the worker process. I looked to delayed_job for how to test this but there are now tests for that portion of the code. Below is the code I need to test. ideas? (I'm using rspec BTW)
def start
say "*** Starting job worker #{#name}"
t = Thread.new do
loop do
delay = Update.work_off(self) #this method well tested
break if $exit
sleep delay
break if $exit
end
clear_locks
end
trap('TERM') { terminate_with t }
trap('INT') { terminate_with t }
trap('USR1') do
say "Wakeup Signal Caught"
t.run
end
see also this thread
The best approach, I believe, is to stub the Thread.new method, and make sure that any "complicated" stuff is in it's own method which can be tested individually. Thus you would have something like this:
class Foo
def start
Thread.new do
do_something
end
end
def do_something
loop do
foo.bar(bar.foo)
end
end
end
Then you would test like this:
describe Foo
it "starts thread running do_something" do
f = Foo.new
expect(Thread).to receive(:new).and_yield
expect(f).to receive(:do_something)
f.start
end
it "do_something loops with and calls foo.bar with bar.foo" do
f = Foo.new
expect(f).to receive(:loop).and_yield #for multiple yields: receive(:loop).and_yield.and_yield.and_yield...
expect(foo).to receive(:bar).with(bar.foo)
f.do_something
end
end
This way you don't have to hax around so much to get the desired result.
You could start the worker as a subprocess when testing, waiting for it to fully start, and then check the output / send signals to it.
I suspect you can pick up quite a few concrete testing ideas in this area from the Unicorn project.
Its impossible to test threads completely. Best you can do is to use mocks.
(something like)
object.should_recieve(:trap).with('TERM').and yield
object.start
How about just having the thread yield right in your test.
Thread.stub(:new).and_yield
start
# assertions...