How to auto save state of long iteration in ruby - ruby

Question

Say I have a program with a runtime in the order of weeks with a structure like this:
(1..1000).each do |number|
('a'..'z').each do |letter|
%w(alpha beta omega whatever foo bar).each do |word|
do_long_running_calculation(number,letter,word)
end
end
end
Since the machine running the program may have a sudden unexpected halt, I'd like to save the index that it was on for each array to a file, such that it can re-start from where it left off instead of starting from the beginning in case of a sudden program abort.
Ultimately if this doesn't yet exist as a library (or easy solution that has evaded me), I'm going to make it myself and post it as an answer, but I would like to avoid re-inventing the wheel

Answer 1

Here's a way to deal with your specific example, which could be generalized.
First, two helpers:
class Range
def size # overwrite
case first
when Fixnum
last - first + 1
when String
last.ord - first.ord + 1
end
end
def [](offset)
case first
when Fixnum
first + offset
when String
(first.ord + offset).chr
end
end
end
For example:
(1..10).size #=> 10
('a'..'z').size #=> 26
(1..10)[4] #=> 5
('a'..'z')[4] #=> "e"
For the example:
loops = [(1..1000), ('a'..'z'), %w(alpha beta omega whatever foo bar)]
loop_sizes = loops.map(&:size)
#=> [1000, 26, 6]
prod = 1
tot_nbr_loops, *prod_loop_sizes = (loop_sizes + [1]).reverse.
map { |n| prod = n*prod }.reverse
#=> [156000, 156, 6, 1]
tot_nbr_loops
#=> 156000
prod_loop_sizes
#=> [156, 6, 1]
Using these objects we can create a method that maps a sequence of integers into the triples that are to be enumerated:
def elements(loops, prod_loop_sizes, offset)
loops.zip(prod_loop_sizes).map do |loop, prod|
div, offset = offset.divmod(prod)
loop[div]
end
end
Let's try it:
elements(loops, prod_loop_sizes, 0) #=> [1, "a", "alpha"]
elements(loops, prod_loop_sizes, 5) #=> [1, "a", "bar"]
elements(loops, prod_loop_sizes, 6) #=> [1, "b", "alpha"]
elements(loops, prod_loop_sizes, 155) #=> [1, "z", "bar"]
elements(loops, prod_loop_sizes, 156) #=> [2, "a", "alpha"]
elements(loops, prod_loop_sizes, 156) #=> [2, "a", "alpha"]
elements(loops, prod_loop_sizes, 156) #=> [2, "a", "alpha"]
elements(loops, prod_loop_sizes, 155999) #=> [1000, "z", "bar"]
So now you could write something like this:
save_interval = 10_000
total_number_loops.times do |i|
a,b,c = elements(loops, prod_loop_sizes, i)
# perform calculations with a,b,c
if i % save_interval == 0
<save the value of i and the current state>
<delete the previous saved state>
end
end
One easy way to save to (retrieve from) file most Ruby objects is to use the method Marshal#dump (Marshal#load). (Note the Marshal file format is not guaranteed to remain the same from one Ruby version to the next.)

Answer 2

Ruby provide us the trap method to handle the OS generated SIGNALS. When the system shutdown, then it send the SIGTERM signal. So all we need to handle that signal.
trap("TERM") do
write_data_to_file
end
def write_data_to_file
# code to save data in the file,
end
To read/write from file, cool answer is here:
https://stackoverflow.com/a/4310299/4136098