I have an application that essentially does this:
queue = SizedQueue.new(2)
Thread.new do
conn = OCI8.new(DSN)
cursor = conn.exec([a query])
cursor.fetch { |rec| queue << rec }
queue << :queue_ended
cursor.close
conn.logoff
end.tap { |t| t.abort_on_exception = true }
until a = queue.pop == :queue_ended
do_things(a)
end
The issue comes when do_things raises an error. The thread seems not to abort. The reason I believe it's the oci8 is because we had this code running with a different set of functions/libraries and everything was fine. The other reason i think it is OCI8 because the connection remains open, surely if the thread had aborted the connection would close?
I occasionally have an issue raising a KeyBoardInterrupt in irb with long to generate queries as well.
Apologies if this isn't much to go on, I don't have much to go on - I can't recreate the issue without using thread and as we all know it's fairly difficult to debug.
N.B I am running with ruby OCI8 (most recent version (2.1.5) on rb 1.9.3).
Thanks for your help in advance
Turn out this isn't an issue, or at least doesn't occur in Ruby 2.0.0; I don't know why though.
Related
I'm using the em-ws-client gem, although I think my question is more general than that. I'm trying to send data from outside the EventMachine receive block, but it takes a very long time (~20s) for the data to be sent:
require "em-ws-client"
m = Mutex.new
c = ConditionVariable.new
Thread.new do
EM.run do
#ws = EM::WebSocketClient.new("ws://echo.websocket.org")
#ws.onopen do
puts "connected"
m.synchronize { c.broadcast }
end
#ws.onmessage do |msg, binary|
puts msg
end
end
end
m.synchronize { c.wait(m) }
#ws.send_message "test"
sleep 100
When I put the #ws.send_message "test" directly into the onopen method it works just fine. I don't understand why my version doesn't work. I found this issue in EventMachine, but I'm not sure whether it's related.
Why does it take so long, and how can I fix that?
EventMachine is strictly single threaded and sharing of sockets between threads is not recommended. What you might be seeing here is an issue with the main EventMachine thread being unaware that you've submitted a send_message call and leaving it buffered for an extended period of time.
I'd be very, very careful when using threads with EventMachine. I've seen it malfunction and crash if you hit thread timing or synchronization problems.
I am trying to optimize this piece of ruby code with Thread, which involves a fair bit of IO and network activity. Unfortunately that is not going down too well.
# each host is somewhere in the local network
hosts.each { |host|
# reach out to every host with something for it to do
# wait for host to complete the work and get back
}
My original plan was to wrap the internal of the loop into a new Thread for each iteration. Something like:
# each host is somewhere in the local network
hosts.each { |host|
Thread.new {
# reach out to every host with something for it to do
# wait for host to complete the work and get back
}
}
# join all threads here before main ends
I was hoping that since this I/O bound even without ruby 1.9 I should be able to gain something but nope, nothing. Any ideas how this might be improved?
I'm not sure how many hosts you have, but if it's a lot you may want to try a producer/consumer model with a fixed number of threads:
require 'thread'
THREADS_COUNT = (ARGV[0] || 4).to_i
$q = Queue.new
hosts.each { |h| $q << h }
threads = (1..THREADS_COUNT).map {
Thread.new {
begin
loop {
host = $q.shift(true)
# do something with host
}
rescue ThreadError
# queue is empty, pass
end
}
}
threads.each(&:join)
If this all too complicated, you may try using xargs -P. ;)
#Fanatic23, before you draw any conclusions, instrument your code with puts and see whether the network requests are actually overlapping. In each call to puts, print a status string indicating the line which is executing, along with Time.now and Time.now.nsec.
You say "since this [is] I/O bound even without ruby 1.9 I should be able to gain something".
Vain hope. :-)
When a thread in Ruby 1.8 blocks on IO, the entire process has blocked on IO. This is because it's using green theads.
Upgrade to Ruby 1.9, and you'll have access to your platform's native threads implementation. For more, see:
http://en.wikipedia.org/wiki/Green_threads
Enjoy!
What are the consequences, (if any) not calling the conn.logoff() method after the following script when connecting to an Oracle database using the Ruby OCI8 library.
conn = OCI8.new('scott', 'tiger')
num_rows = conn.exec('SELECT * FROM emp') do |r|
puts r.join(',')
end
puts num_rows.to_s + ' rows were processed.'
The reason I'm asking because we're experiencing slow downs with other applications that connect to this same Oracle db.
Thanks
I would imagine that when the Ruby process exits, the session will be killed automatically.
You could check by querying v$session to see if the ruby process is still connected to Oracle after Ruby exits.
Given only the information in your question, its really impossible to say what could be causing slowdowns - there are so many variable.
If you don't call conn.logoff(), the connection is alive even though it is garbage-collected until the ruby process exits.
The problem is fixed in ruby-oci8 2.1, which have not been released yet though.
I think I'm running up against a fundamental misunderstanding on my part of how threading works in ruby and I'm hoping to get some insight.
I'd like to have a simple producer and consumer. First, a producer thread that pulls lines from a file and sticks them into a SizedQueue; when those run out, stick some tokens on the end to let the consumer(s) know things are done.
require 'thread'
numthreads = 2
filename = 'edition-2009-09-11.txt'
bq = SizedQueue.new(4)
producerthread = Thread.new(bq) do |queue|
File.open(filename) do |f|
f.each do |r|
queue << r
end
end
numthreads.times do
queue << :end_of_producer
end
end
Now a few consumers. For simplicity, let's have them do nothing.
consumerthreads = []
numthreads.times do
consumerthreads << Thread.new(bq) do |queue|
until (line = queue.pop) === :end_of_producer
# do stuff in here
end
end
end
producerthread.join
consumerthreads.each {|t| t.join}
puts "All done"
My understanding is that (a) the producer thread will block once the SizedQueue is full and eventually get back to filling it up, and (b) the consumer threads will pull from the SizedQueue, blocking when it empties, and eventually finish.
But under ruby1.9 (ruby 1.9.1p243 (2009-07-16 revision 24175) [i386-darwin9]) I get a deadlock error on the joins. What's going on here? I just don't see where there's any interaction between the threads except via the SizedQueue,which is supposed to be thread-safe.
Any insight would be much-appreciated.
Your understanding is correct and your code works on my machine, on a slightly newer version of Ruby (both ruby 1.9.2dev (2009-08-30 trunk 24705) [i386-darwin10.0.0] and ruby 1.9.2dev (2009-08-30 trunk 24705) [i386-darwin10.0.0])
I have a library method that occasionally hangs on a network connection, and there's no timeout mechanism.
What's the easiest way to add my own? Basically, I'm trying to keep my code from getting indefinitely stuck.
timeout.rb has some problems where basically it doesn't always work quite right, and I wouldn't recommend using it. Check System Timer or Terminator instead
The System Timer page in particular describes why timeout.rb can fail, complete with pretty pictures and everything. Bottom line is:
For timeout.rb to work, a freshly created “homicidal” Ruby thread has to be scheduled by the Ruby interpreter.
M.R.I. 1.8, the interpreter used by most Ruby applications in production, implements Ruby threads as green threads.
It is a well-known limitations of the green threads (running on top of a single native thread) that when a green thread performs a blocking system call to the underlying operating systems, none of green threads in the virtual machine will run until the system call returns.
Answered my own question:
http://www.ruby-doc.org/stdlib/libdoc/timeout/rdoc/index.html
require 'timeout'
status = Timeout::timeout(5) {
# Something that should be interrupted if it takes too much time...
}
To prevent an ugly error on timeout I suggest enclosing it and using a rescue like this:
begin
status = Timeout::timeout(5) do
#Some stuff that should be interrupted if it takes too long
end
rescue Timeout::Error
puts "Error message: It took too long!\n"
end