Ruby : The best way to manage a large 3d array - ruby

Question

I would like to know what is the best way to manage a large 3d array with something like :
x = 1000
y = 1000
z = 100
=> 100000000 objects
And each cell is an object with some amount of data.
Simple methods are very loooooong even if all data are collapsed (I a first tryed an array of array of array of objects)
class Test
def initialize
#name = "Test"
end
end
qtt = 1000*1000*100
Array.new(qtt).each { |e| e = Test.new }
I read somewhere that DB could be a good thing for such cases.
What do you think about this ?
What am I trying to do ?
This "matrix" represents a world. And each element is a 1mx1mx2m block who could be a different kind (water, mud, stone, ...) Some block could be empty too.
But the user should be able to remove blocks everywhere and change everything around (if they where water behind, it will flow through the hole for exemple.
In fact what I wish to do is not Minecraft be a really small clone of DwarfFortress (http://www.bay12games.com/dwarves/)
Other interesting things
In my model the groud is at level 10. It means that [0,10] is empty sky in most of cases.
Only hills and parts of mountains could be present on those layers.
Underground is basicaly unknown and not dug. So we should not have to add instances for unused blocks.
What we should add from the beginning to the model : gems, gold, water who could stored without having to store the adjacent stone/mood/earth blocks.
At the beginning of the game, 80% of the cube doesn't need to be loaded in memory.
Each time we dig we create new blocks : the empty block we dug and the blocks around.
The only things we should index is :
underground rivers
underground lakes
lava rivers

Answer 1

Holding that many objects in memory is never a good thing. A flat-file or database-centric approach would be a lot more efficient and easier to maintain.
What I would do - The object-oriented approach
Store the parameters of the blocks as simple data and construct the objects dynamically.
Create a Block class to represent a block in the game, and give it variables to hold the parameters of that particular block:
class Block
# location of the Block
attr_accessor :x, :y, :z
# an individual id for the Block
attr_accessor :id
# to define the block type (rock, water etc.)
attr_accessor :block_type
# and add any other attributes of a Block...
end
I'd then create a few methods that would enable me to serialise/de-serialise the data to a file or database.
As you've stated it works on a board, you'd also need a Board class to represent it that would maintain the state of the game as well as perform actions on the Block objects. Using the x, y, z attributes from each Block you can determine its location within the game. Using this information you can then write a method in the Block class that locates those blocks adjacent to the current one. This would enable you to perform the "cascading" effects you talk about where one Block is affected by actions on another.
Accessing the data efficiently
This will rely entirely on how you choose to serialise the Block objects. I would probably choose a binary format to reduce unnecessary data reads and store the objects via their id parameter, and then use something like MMIO to quickly do random-access reads/writes on a large data file in an Array-like manner. This will allow you to access the data quickly and efficiently, without the memory overhead. How you read the data will relate to your adjacent blocks method above.
You can of course also choose the DB storage route which will allow you to isolate the Blocks and do lookups on particular blocks in a higher-level manner, however that might give you a bit of extra overhead.
It sounds like an interesting project, I hope this helps a bit! :)
P.S With regards to the comment above by #Linuxious about choosing a different language. Yes this might be true in some cases, but a skilled programmer never blames his tools. A program is only as efficient as the programmer makes it...unless you're writing it in Java ;)