Cumulative Maxima as Indicated by X in APL - algorithm

Question

The third item in the FinnAPL Library is called “Cumulative maxima (⌈) of subvectors of Y indicated by X ” where X is a binary vector and Y os a vector of numbers. Here's an example of its usage:
X←1 0 0 0 1 0 0 0
Y←9 78 3 2 50 7 69 22
Y[A⍳⌈\A←⍋A[⍋(+\X)[A←⍋Y]]] ⍝ output 9 78 78 78 50 50 69 69
You can see that beginning from either the beginning or from any 1 value in the X array, the cumulave maximum is found for all corresponding digits in Y until another 1 is found in X. In the example given, X is divding the array into two equal parts of 4 numbers each. In the first part, 9 is the maxima until 78 is encountered, and in the second part 50 is the maxima until 69 is encountered.
That's easy enough to understand, and I could blindly use it as is, but I'd like to understand how it works, because APL idioms are essentially algorithms made up of operators and functions. To understand APL well, it's important to understand how the masters were able to weave it all together into such compact and elegant lines of code.
I find this particular idiom especially hard to understand because of the indexing nested two layers deep. So my question is, what makes this idiom tick?

Answer 1

This idiom can be broken down into smaller idioms, and most importantly, it contains idiom #11 from the FinnAPL Library entitled:
Grade up (⍋) for sorting subvectors of Y indicated by X
Using the same values for X and Y given in the question, here's an example of its usage:
X←1 0 0 0 1 0 0 0
Y←9 78 3 2 50 7 69 22
A[⍋(+\X)[A←⍋Y]] ⍝ output 4 3 1 2 6 8 5 7
As before, X is dividing the vector into two halves, and the output indicates, for each position, what digit of Y is needed to sort each of the halves. So, the 4 in the output is saying that it needs the 4th digit of Y (2) in the 1st position; the 3 indicates the 3rd digit (3) in the 2nd position; the 1 indicates the 1st digit (9) in the third position; etc. Thus, if we apply this indexing to Y, we get:
Y[A[⍋(+\X)[A←⍋Y]]] ⍝ output 2 3 9 78 7 22 50 69
In order to understand the indexing within this grade-up idiom, consider what is happening with the following:
(+\X)[A←⍋Y] ⍝ Sorted Cumulative Addition
Breaking it down step by step:
A←⍋Y ⍝ 4 3 6 1 8 5 7 2
+\X ⍝ 1 1 1 1 2 2 2 2
(+\X)[A←⍋Y] ⍝ 1 1 2 1 2 2 2 1 SCA
A[⍋(+\X)[A←⍋Y]] ⍝ 4 3 1 2 6 8 5 7
You can see that sorted cumulative addition (SCA) of X 1 1 2 1 2 2 2 1 applied to A acts as a combination of compress left and compress right. All values of A that line up with a 1 are moved to the left, and those lining up with a 2 move to the right. Of course, if X had more 1s, it would be compressing and locating the compressed packets in the order indicated by the values of the SCA result. For example, if the SCA of X were like 3 3 2 1 2 2 1 1 1, you would end up with the 4 digits corresponding to the 1s, followed by the 3 digits corresponding to the 2s, and finally, the 2 digits corresponding to the 3s.
You may have noticed that I skipped the step that would show the effect of grade up ⍋:
(+\X)[A←⍋Y] ⍝ 1 1 2 1 2 2 2 1 SCA
⍋(+\X)[A←⍋Y] ⍝ 1 2 4 8 3 5 6 7 Grade up
A[⍋(+\X)[A←⍋Y]] ⍝ 4 3 1 2 6 8 5 7
The effect of compression and rearrangement isn't accomplised by SCA alone. It effectively acts as rank, as I discussed in another post. Also in that post, I talked about how rank and index are essentially two sides of the same coin, and you can use grade up to switch between the two. Therefore, that is what is happening here: SCA is being converted to an index to apply to A, and the effect is grade-up sorted subvectors as indicated by X.
From Sorted Subvectors to Cumulative Maxima
As already described, the result of sorting the subvectors is an index, which when applied to Y, compresses the data into packets and arranges those packets according to X. The point is that it is an index, and once again, grade up is applied, which converts indexes into ranks:
⍋A[⍋(+\X)[A←⍋Y]] ⍝ 3 4 2 1 7 5 8 6
The question here is, why? Well, the next step is applying a cumulative maxima, and that really only makes sense if it is applied to values for rank which represent relative magnitude within each packet. Looking at the values, you can see that 4 is is the maxima for the first group of 4, and 8 is for the second group. Those values correspond to the input values of 78 and 69, which is what we want. It doesn't make sense (at least in this case) to apply a maxima to index values, which represent position, so the conversion to rank is necessary. Applying the cumulative maxima gives:
⌈\A←⍋A[⍋(+\X)[A←⍋Y]] ⍝ 3 4 4 4 7 7 8 8
That leaves one last step to finish the index. After doing a cumulative maxima operation, the vector values still represent rank, so they need to be converted back to index values. To do that, the index-of operator is used. It takes the value in the right argument and returns their position as found in the left argument:
A⍳⌈\A←⍋A[⍋(+\X)[A←⍋Y]] ⍝ 1 2 2 2 5 5 7 7
To make it easier to see:
3 4 2 1 7 5 8 6 left argument
3 4 4 4 7 7 8 8 right argument
1 2 2 2 5 5 7 7 result
The 4 is in the 2nd position in the left argument, so the result shows a 2 for every 4 in the right argument. The index is complete, so applying it to Y, we get the expected result:
Y[A⍳⌈\A←⍋A[⍋(+\X)[A←⍋Y]]] ⍝ 9 78 78 78 50 50 69 69

Answer 2

My implementation:
X←1 0 0 0 1 0 0 0
Y←9 78 3 2 50 7 69 22
¯1+X/⍳⍴X ⍝ position
0 4
(,¨¯1+X/⍳⍴X)↓¨⊂Y
9 78 3 2 50 7 69 22 50 7 69 22
(1↓(X,1)/⍳⍴X,1)-X/⍳⍴X ⍝ length
4 4
(,¨(1↓(X,1)/⍳⍴X,1)-X/⍳⍴X)↑¨(,¨¯1+X/⍳⍴X)↓¨⊂Y
9 78 3 2 50 7 69 22
⌈\¨(,¨(1↓(X,1)/⍳⍴X,1)-X/⍳⍴X)↑¨(,¨¯1+X/⍳⍴X)↓¨⊂Y
9 78 78 78 50 50 69 69
∊⌈\¨(,¨(1↓(X,1)/⍳⍴X,1)-X/⍳⍴X)↑¨(,¨¯1+X/⍳⍴X)↓¨⊂Y
9 78 78 78 50 50 69 69
Have a nice day.