Linearly reading a multi-dimensional array obeying dimensional sub-sectioning - algorithm

Question

I have an API for reading multi-dimensional arrays, requiring to pass a vector of ranges to read sub-rectangles (or hypercubes) from the backing array. I want to read this array "linearly", all elements in some given order with arbitrary chunk sizes. Thus, the task is with an off and a len, to translate the elements covered by this range into the smallest possible set of hyper-cubes, i.e. the smallest number of read commands issued in the API.
For example, we can calculate index vectors for the set of dimensions giving a linear index:
def calcIndices(off: Int, shape: Vector[Int]): Vector[Int] = {
val modsDivs = shape zip shape.scanRight(1)(_ * _).tail
modsDivs.map { case (mod, div) =>
(off / div) % mod
}
}
Let's say the shape is this, representing an array with rank 4 and 120 elements in total:
val sz = Vector(2, 3, 4, 5)
val num = sz.product // 120
A utility to print these index vectors for a range of linear offsets:
def printIndices(off: Int, len: Int): Unit =
(off until (off + len)).map(calcIndices(_, sz))
.map(_.mkString("[", ", ", "]")).foreach(println)
We can generate all those vectors:
printIndices(0, num)
[0, 0, 0, 0]
[0, 0, 0, 1]
[0, 0, 0, 2]
[0, 0, 0, 3]
[0, 0, 0, 4]
[0, 0, 1, 0]
[0, 0, 1, 1]
[0, 0, 1, 2]
[0, 0, 1, 3]
[0, 0, 1, 4]
[0, 0, 2, 0]
[0, 0, 2, 1]
[0, 0, 2, 2]
[0, 0, 2, 3]
[0, 0, 2, 4]
[0, 0, 3, 0]
[0, 0, 3, 1]
[0, 0, 3, 2]
[0, 0, 3, 3]
[0, 0, 3, 4]
[0, 1, 0, 0]
...
[1, 2, 1, 4]
[1, 2, 2, 0]
[1, 2, 2, 1]
[1, 2, 2, 2]
[1, 2, 2, 3]
[1, 2, 2, 4]
[1, 2, 3, 0]
[1, 2, 3, 1]
[1, 2, 3, 2]
[1, 2, 3, 3]
[1, 2, 3, 4]
Let's look at an example chunk that should be read,
the first six elements:
val off1 = 0
val len1 = 6
printIndices(off1, len1)
I will already partition the output by hand into hypercubes:
// first hypercube or read
[0, 0, 0, 0]
[0, 0, 0, 1]
[0, 0, 0, 2]
[0, 0, 0, 3]
[0, 0, 0, 4]
// second hypercube or read
[0, 0, 1, 0]
So the task is to define a method
def partition(shape: Vector[Int], off: Int, len: Int): List[Vector[Range]]
which outputs the correct list and uses the smallest possible list size.
So for off1 and len1, we have the expected result:
val res1 = List(
Vector(0 to 0, 0 to 0, 0 to 0, 0 to 4),
Vector(0 to 0, 0 to 0, 1 to 1, 0 to 0)
)
assert(res1.map(_.map(_.size).product).sum == len1)
A second example, elements at indices 6 until 22, with manual partitioning giving three hypercubes or read commands:
val off2 = 6
val len2 = 16
printIndices(off2, len2)
// first hypercube or read
[0, 0, 1, 1]
[0, 0, 1, 2]
[0, 0, 1, 3]
[0, 0, 1, 4]
// second hypercube or read
[0, 0, 2, 0]
[0, 0, 2, 1]
[0, 0, 2, 2]
[0, 0, 2, 3]
[0, 0, 2, 4]
[0, 0, 3, 0]
[0, 0, 3, 1]
[0, 0, 3, 2]
[0, 0, 3, 3]
[0, 0, 3, 4]
// third hypercube or read
[0, 1, 0, 0]
[0, 1, 0, 1]
expected result:
val res2 = List(
Vector(0 to 0, 0 to 0, 1 to 1, 1 to 4),
Vector(0 to 0, 0 to 0, 2 to 3, 0 to 4),
Vector(0 to 0, 1 to 1, 0 to 0, 0 to 1)
)
assert(res2.map(_.map(_.size).product).sum == len2)
Note that for val off3 = 6; val len3 = 21, we would need four readings.

Answer 1

The idea of the following algorithm is as follows:
a point-of-interest (poi) is the left-most position
at which two index representations differ
(for example for [0, 0, 0, 1] and [0, 1, 0, 0] the poi is 1)
we recursively sub-divide the original (start, stop) linear index range
we use motions in two directions, first by keeping the start constant
and decreasing the stop through a special "ceil" operation on the start,
later by keeping the stop constant and increasing the start through
a special "floor" operation on the stop
for each sub range, we calculate the poi of the boundaries, and
we calculate "trunc" which is ceil or floor operation described above
if this trunc value is identical to its input, we add the entire region
and return
otherwise we recurse
the special "ceil" operation takes the previous start value and
increases the element at the poi index and zeroes the subsequent elements;
e.g. for [0, 0, 1, 1] and poi = 2, the ceil would be [0, 0, 2, 0]
the special "floor" operation takes the previous stop value and
zeroes the elements after the poi index;
e.g. for [0, 0, 1, 1], and poi = 2, the floor would be [0, 0, 1, 0]
Here is my implementation. First, a few utility functions:
def calcIndices(off: Int, shape: Vector[Int]): Vector[Int] = {
val modsDivs = (shape, shape.scanRight(1)(_ * _).tail, shape.indices).zipped
modsDivs.map { case (mod, div, idx) =>
val x = off / div
if (idx == 0) x else x % mod
}
}
def calcPOI(a: Vector[Int], b: Vector[Int], min: Int): Int = {
val res = (a.drop(min) zip b.drop(min)).indexWhere { case (ai,bi) => ai != bi }
if (res < 0) a.size else res + min
}
def zipToRange(a: Vector[Int], b: Vector[Int]): Vector[Range] =
(a, b).zipped.map { (ai, bi) =>
require (ai <= bi)
ai to bi
}
def calcOff(a: Vector[Int], shape: Vector[Int]): Int = {
val divs = shape.scanRight(1)(_ * _).tail
(a, divs).zipped.map(_ * _).sum
}
def indexTrunc(a: Vector[Int], poi: Int, inc: Boolean): Vector[Int] =
a.zipWithIndex.map { case (ai, i) =>
if (i < poi) ai
else if (i > poi) 0
else if (inc) ai + 1
else ai
}
Then the actual algorithm:
def partition(shape: Vector[Int], off: Int, len: Int): List[Vector[Range]] = {
val rankM = shape.size - 1
def loop(start: Int, stop: Int, poiMin: Int, dir: Boolean,
res0: List[Vector[Range]]): List[Vector[Range]] =
if (start == stop) res0 else {
val last = stop - 1
val s0 = calcIndices(start, shape)
val s1 = calcIndices(stop , shape)
val s1m = calcIndices(last , shape)
val poi = calcPOI(s0, s1m, poiMin)
val ti = if (dir) s0 else s1
val to = if (dir) s1 else s0
val st = if (poi >= rankM) to else indexTrunc(ti, poi, inc = dir)
val trunc = calcOff(st, shape)
val split = trunc != (if (dir) stop else start)
if (split) {
if (dir) {
val res1 = loop(start, trunc, poiMin = poi+1, dir = true , res0 = res0)
loop (trunc, stop , poiMin = 0 , dir = false, res0 = res1)
} else {
val s1tm = calcIndices(trunc - 1, shape)
val res1 = zipToRange(s0, s1tm) :: res0
loop (trunc, stop , poiMin = poi+1, dir = false, res0 = res1)
}
} else {
zipToRange(s0, s1m) :: res0
}
}
loop(off, off + len, poiMin = 0, dir = true, res0 = Nil).reverse
}
Examples:
val sz = Vector(2, 3, 4, 5)
partition(sz, 0, 6)
// result:
List(
Vector(0 to 0, 0 to 0, 0 to 0, 0 to 4), // first hypercube
Vector(0 to 0, 0 to 0, 1 to 1, 0 to 0) // second hypercube
)
partition(sz, 6, 21)
// result:
List(
Vector(0 to 0, 0 to 0, 1 to 1, 1 to 4), // first read
Vector(0 to 0, 0 to 0, 2 to 3, 0 to 4), // second read
Vector(0 to 0, 1 to 1, 0 to 0, 0 to 4), // third read
Vector(0 to 0, 1 to 1, 1 to 1, 0 to 1) // fourth read
)
The maximum number of reads, if I'm not mistaken, would be 2 * rank.