How do Enumerators work in Ruby 1.9.1? - ruby

Question

This question is not about how to use Enumerators in Ruby 1.9.1 but rather I am curious how they work. Here is some code:
class Bunk
def initialize
#h = [*1..100]
end
def each
if !block_given?
enum_for(:each)
else
0.upto(#h.length) { |i|
yield #h[i]
}
end
end
end
In the above code I can use e = Bunk.new.each, and then e.next, e.next to get each successive element, but how exactly is it suspending execution and then resuming at the right spot?
I am aware that if the yield in the 0.upto is replaced with Fiber.yield then it's easy to understand, but that is not the case here. It is a plain old yield, so how does it work?
I looked at enumerator.c but it's neigh on incomprehensible for me. Maybe someone could provide an implementation in Ruby, using fibers, not 1.8.6 style continuation-based enumerators, that makes it all clear?

Answer 1

Here's a plain ruby enumerator that uses Fibers and should pretty much behave like the original:
class MyEnumerator
include Enumerable
def initialize(obj, iterator_method)
#f = Fiber.new do
obj.send(iterator_method) do |*args|
Fiber.yield(*args)
end
raise StopIteration
end
end
def next
#f.resume
end
def each
loop do
yield self.next
end
rescue StopIteration
self
end
end
And in case anyone is feeling uneasy about using exceptions for control flow: The real Enumerator raises StopIteration at the end, too, so I just emulated the original behaviour.
Usage:
>> enum = MyEnumerator.new([1,2,3,4], :each_with_index)
=> #<MyEnumerator:0x9d184f0 #f=#<Fiber:0x9d184dc>
>> enum.next
=> [1, 0]
>> enum.next
=> [2, 1]
>> enum.to_a
=> [[3, 2], [4, 3]]

Answer 2

Actually in your e = Bunk.new.each the else clause is not executed initially. Instead the 'if !block_given' clause executes and returns an enumerator object. The enumerator object does keep a fiber object internally. (At least that is what it looks like in enumerator.c)
When you call e.each it is calling a method on an enumerator which uses a fiber internally to keep track of its execution context. This method calls the Bunk.each method using the fibers execution context. The Bunk.each call here does execut the else clause and yields up the value.
I do not know how yield is implemented or how a fiber tracks the execution context. I haven't looked at that code. Almost all of the enumerator and fiber magic is implemented in C.
Are you really asking how fibers and yield are implemented? What level of detail are you looking for?
If I am off base please correct me.

Answer 3

As the other posters noted, I believe it creates its own private "fiber" [in 1.9]. In 1.8.7 (or 1.8.6 if you use the backports gem) somehow or other it does the same thing (perhaps because all threads in 1.8 are the equivalent of fibers, it just uses them?)
Thus in 1.9 and 1.8.x, if you chain several of them together
a.each_line.map.each_with_index { }
It actually flows through that whole chain with each line, kind of like a pipe on the command line
http://pragdave.blogs.pragprog.com/pragdave/2007/12/pipelines-using.html
HTH.

Answer 4

I think this would be more accurate. Calling each on the enumerator should be the same as calling the original iterator method. So I would slightly change the original solution to this:
class MyEnumerator
include Enumerable
def initialize(obj, iterator_method)
#f = Fiber.new do
#result = obj.send(iterator_method) do |*args|
Fiber.yield(*args)
end
raise StopIteration
end
end
def next(result)
#f.resume result
end
def each
result = nil
loop do
result = yield(self.next(result))
end
#result
end
end