I've been playing around with threads but I keep running into a problem where the treads seem to just die or stop.
Whats going on here? And how do I get round it?
I've included the code, but didn't paste it here as I think this problem is more fundamental to ruby.
source code
thanks.
Edit
Ruby 1.8, MacOS (snow leopard)
It looks like you're forgetting to add your new Thread objects to your threads object.
3.times do |t|
threads << Thread.new { word_list.process }
end
Your threads.each {|t| t.join} is working on an empty array, and so is ignoring the threads you did create. Make the change and it should wait.
Edit: I meant to << onto the array, not set it equal.
If you've got threads randomly "going away", make sure you've set Thread.abort_on_exception to true. That'll stop the interpreter if an uncaught exception reaches the top-level in a background thread (via SystemExit, so don't rescue Exception or they'll get swallowed), and can be immensely useful in tracking down random bugs.
Related
I have a pretty large ruby (non-rails) application that I'm developing. It's reasonably fast considering how large and complex it is (go ruby!), but sometimes I fat finger a method name and get the NoMethodError.
And usually when this happens, the application hangs for like 20 to 30 seconds to just print out the backtrace.
Specifically, if I do something like this:
puts "about to crash!"
Array.new().inspekt # NoMethodError here
I see the "about to crash!" right away, and then 20s or so nothing seems to happen before I finally get the NoMethodError and backtrace.
At first I thought it might be the "did you mean" gem, so I turned that off with --disable-did_you_mean on the command line, and that turned off the "did you mean" suggestions, but nothing sped up the backtrace.
What's interesting is that this is only for NoMethodError.
If I cause some other exception, such as:
puts "about to crash!"
a = 3/0
Then I see the backtrace immediately.
And to make things even weirder, if I interrupt the process right after the "about to crash!" (such as with a ctrl-c on unix) then I immediately get the NoMethodError and it's backtrace. So it has the information - but ruby is stuck on trying to clean something up perhaps, something that only gets cleaned up on NoMethodError?
Info: ruby 2.7.0
OS: CentOS Linux release 7.5.1804
UPDATE - to responses so far:
Everyone seems to be concerned about the backtrace and profiling the ruby code.
Except the slowdown is NOT happening there. There are NO LINES OF RUBY CODE that are executed during the slowdown. All of the lines prior to this, "in the backtrace" are already executed and in a matter of a second or so. Then the system hangs, between the puts and the NoMethodError. There is no ruby code in between to profile, so any profiler that is looking at code written in my ruby script isn't going to help. The slowdown is something internal to ruby and is not in my code, unless I'm terribly confused about what's happening.
To be very clear:
Line 10042: puts "HERE" # Happens at ~1s
Line 10043: Array.new().inspekt # Happens at ~20-30s
There is no code between those lines to profile. The 20-30s is not happening in any code before line 10042 executes, so profiling that will not help.
I do have other Fibers that are paused. But there is no code here that yields to them. Is it possible that there's some strange built-in yield code that attempts to run other (paused) fibers when an exception is hit? I can't think of a reason you'd ever want this behavior, and many reasons why it would be catastrophic, but I can't think of anything else that would cause this problem (that is also killable with a ctrl-c!)
I would try to debug the full backtrace in there to see what is actually happening
begin
puts "about to crash!"
Array.new().inspekt
rescue => e
puts e.backtrace
raise # raise anyway
end
In my case I get 20 lines of backtrace with ruby 2.6.3 and irb, if that doesn't really tell you anything interesting I would then do the tedious work of measuring each runtime by modifying each file of the backtrace and printing the times at each step, debugging yay!
Working with ruby 1.9.3 with tk and I've found that I can't do a fork inside the mainloop that calls "exit" - I need to get out of the fork by doing something like exec().
Example program:
#!/usr/bin/ruby
require 'tk'
root = TkRoot.new
def doit
unless fork
puts "Inside the fork"
exit # This is where it falls apart
end
end
TkButton.new(root) {
text 'go'
command proc { doit }
}.pack
Tk.mainloop()
Press the button and we properly fork, but when fork calls 'exit' then we get:
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
ruby: ../../src/xcb_io.c:274: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
ruby: ../../src/xcb_io.c:274: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
I've discovered that I can hack a workaround by using something like "exec('echo')" instead of the exit, but that's just plain silly. What's going on that exit doesn't work? Is there some way to call XInitThreads from ruby (this isn't jruby or fxruby) that I can use and will that help?
From my research so far, it seems that ruby and Tk are pretty broken the moment you introduce threads or forking, but I have not been able to find a clean way to deal with this problem.
This problem exists with any Tcl built with thread support.
You will have to restructure your program so that the Tcl/Tk processes are separate entities and do not get forked.
The gui will need to send any input commands to the appropriate ruby thread and the threads will need to send data to the gui to be displayed.
I've been learning Ruby for the past few days, and I've run into a few issues concerning the implementation of threads. I've programmed in other languages before (mainly Java and C), and I still couldn't figure out what the issue is. I'm running ruby 2.1.2p95 on Ubuntu Server 14.10. The code in question is from Mr. Neighborly's Humble Little Ruby Book:
mate = Thread.new do
puts "Ahoy! Can I be dropping the anchor sir?"
Thread.stop
puts "Aye sir, dropping anchor!"
end
Thread.pass
puts "CAPTAIN: Aye, laddy!"
mate.run
mate.join
The output should be:
Ahoy! Can I be dropping the anchor sir?
CAPTAIN: Aye, laddy!
Aye sir, dropping anchor!
But instead, I'm receiving the following join and deadlock error:
CAPTAIN: Aye, laddy!
Ahoy! Can I be dropping the anchor sir?
ex2.rb:12:in `join': No live threads left. Deadlock? (fatal)
from ex2.rb:12:in `<main>'
I've run into errors with other threading examples from other resources as well, and have tried running the examples on other Ubuntu machines as well as trying Ruby 2.2. Is there a blatant concept that I'm missing out on? Has something changed in recent revisions of Ruby that would deem the examples out-of-date? Thank you for your help!
Has something changed in recent revisions of Ruby that would deem the examples out-of-date?
Yes. It looks like this book was written for Ruby 1.8, which used green threads. Ruby 1.9 onwards uses native threads (where the threads are scheduled by the OS).
Compare the documentation for the Thread.pass method in Ruby 1.8.7:
Invokes the thread scheduler to pass execution to another thread.
In Ruby 2.1.2 (the version you are using), this methods documentation looks like this:
Give the thread scheduler a hint to pass execution to another thread. A running thread may or may not switch, it depends on OS and processor.
So in current versions the scheduling is not deterministic in the way it was in Ruby 1.8.7, the OS is free to ignore the call to Thread.pass and run the main thread first, which causes the problems.
Running this script on my machine (Mac OS 10.9, Ruby 2.2.0) I get both results, sometimes it works and I see:
Ahoy! Can I be dropping the anchor sir?
CAPTAIN: Aye, laddy!
Aye sir, dropping anchor!
Other times it fails with:
CAPTAIN: Aye, laddy!
Ahoy! Can I be dropping the anchor sir?
capt-thread.rb:12:in `join': No live threads left. Deadlock? (fatal)
from capt-thread.rb:12:in `<main>'
According to GIL implementation in Ruby (MRI), the code below must fail by printing a message more than one time. But it doesn't, it always print it one time:
class Sheep
def initialize
#shorn = false
end
def shorn?
#shorn
end
def shorn!
puts "shearing..."
#shorn = true
end
end
s = Sheep.new
55.times.map do
Thread.new { s.shorn! unless s.shorn? }
end.each(&:join)
How come?
$ ruby --version
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]
It depends a bit on which exact ruby version you use (which differ in the way they schedule threads). On my system it depends a bit on the overall system load and how fast the terminal feels, but on Ruby 2.0.00p481 I get between 1 and 55 lines of output, on Ruby 1.8.7, I consistently get only one line.
It should be noted here that Ruby 2.0 and higher uses actual OS threads (albeit still with a GIL) while Ruby 1.8 uses internal green threads with its own scheduling. It might be very well possible that older ruby versions schedule threads more granular.
In any case, you should not rely on any incidentally thread scheduling behavior. This is not part of any documented behavior and things will change on different systems as as Ruby matures. You should always ensure that you use shared data structures safely when using threads.
I use Ruby version ruby 2.1.5p273 and I suppose your slightly different Ruby version should yield similar results.
I have different results every time I run the program.
I tried with one core enabled and fore cores enabled. I don't see a difference. It is not thread safe, as you expected.
Otherwise the only answer I can come up with is that your program is too fast/lightweight, so that the interpreter does not think of thread switching too often.
I have only one suggestion in this case. A trick you could use to give the interpreter a hint that maybe she could switch threads. You could use the sleep function.
In your example I would put it just before the race condition:
def shorn!
sleep 0.0001
puts "shearing..."
#shorn = true
end
If you'd like to have more info about the GIL I can recommend Jesse Storimer's Nobody understands the GIL
If you'd like to read more about Ruby and concurrency I can recommend Dotan Nahum's Pragmatic Concurrency with Ruby
The trick I suggested was mentioned in this answer
As others have mentioned, the GIL's behavior is not documented and is totally implementation-dependent. You shouldn't rely on any expectations about its scheduling behavior.
A more detailed (and also more general) answer, however, is that the scheduler switches execution between threads to make sure that no single thread blocks the process. This switch is called a context switch or more specifically a thread switch.
When the context switch occurs, the current thread's execution is paused and another thread's execution is resumed. If it's a brand new thread that's being "resumed," then it means that the new thread's execution starts from the beginning.
In the case of your program, each new thread begins with
s.shorn?
as it evaluates unless s.shorn?. At this point, #shorn == false and s.shorn? evaluates to false. So then the thread runs:
s.shorn!
The first command in #shorn! that gets run is:
puts "shearing..."
What happens next depends on the thread scheduler:
If the scheduler decides to let the current thread continue executing, then the next command that gets executed is #shorn = true. Then the thread ends, the scheduler starts the next thread, unless s.shorn? evaluates to true, and the thread stops. This behavior repeats in a loop until there are no more threads left.
If the scheduler decides to switch to another thread, then it will pause execution right before #shorn = true and start running the same code as before from the beginning. That means that #shorn == false when the new thread starts, and so puts "shearing..." will execute again.
As you can see, it all depends on when the scheduler decides to perform a context switch.
But what about the GIL?
The GIL is a horribly misunderstood part of MRI Ruby. There are plenty of resources out there to explain how the GIL works, but in this case the most important thing that you should know is that the GIL doesn't guarantee that each thread will run sequentially.
Instead, the GIL merely guarantees that most core Ruby methods that are implemented in C (for example, Array#<<) won't be interrupted by a context switch until they are finished. In the case of puts "shearing...", I haven't looked at the code for puts, but probably the GIL guarantees that no other thread will run until the currently running thread finishes executing puts.
As for why when you ran your code under MRI 1.8.7 it only displayed shearing... once, that doesn't necessarily have anything to do with green vs. native threads. The better answer is that it was a coincidence. The more precise answer is that in your case, for some reason the scheduler decided to interrupt the first thread after running #shorn = true. This behavior may possibly have been due to green threads in the sense that maybe your native scheduler interrupts more frequently than Ruby's scheduler (hence the "more granular" suggestion in one of the answers below), but that's not necessarily true. It could also have been a fluke.
Multithreading in Ruby is really easy to mess up. Hence why Matz recommends sticking to forking processes, which is memory-inefficient but removes the burden of managing threads. Another approach for larger projects would be to use a library like Celluloid, which abstracts away Ruby's thread safety mechanisms. For a small example like this, however, a simple mutex would do:
semaphore = Mutex.new
s = Sheep.new
55.times.map {
Thread.new {
semaphore.synchronize do
s.shorn! unless s.shorn?
end
}
}.each(&:join)
I'm working with 6 csv files that each contain the attributes of an object. I can read them one at a time, but the idea of splitting each one to a thread to do in parallel is very appealing.
I've created a database object (no relational DBs or ORMs allowed) that has an array for each of the objects it is holding. I've tried the following to make each CSV open and initialize concurrently, but have seen no impact on speed.
threads = []
CLASS_FILES.each do |klass, filename|
threads << Thread.new do
file_to_objects(klass, filename)
end
end
threads.each {|thread| thread.join}
update
end
def self.load(filename)
CSV.open("data/#{filename}", CSV_OPTIONS)
end
def self.file_to_objects(klass, filename)
file = load(filename)
method_name = filename.sub("s.csv","")
file.each do |line|
instance = klass.new(line.to_hash)
Database.instance.send("#{method_name}") << instance
end
end
How can I speed things up in ruby (MRI 1.9.3)? Is this a good case for Rubinius?
Even though Ruby 1.9.3 uses native threads in order to implement concurrency, it has a global interpreter lock which makes sure only one thread executes at a time.
Therefore, nothing really runs in parallel in C Ruby. I know that JRuby imposes no internal lock on any thread, so try using it to run your code, if possible.
This answer by Jörg W Mittag has a more in-depth look at the threading models of the several Ruby implementations. It isn't clear to me whether Rubinius is fit for the job, but I'd give it a try.