Assume, that we have large file, which contains descriptions of the cells of two matrices (A and B):
+---------------------------------+
| i | j | value | matrix |
+---------------------------------+
| 1 | 1 | 10 | A |
| 1 | 2 | 20 | A |
| | | | |
| ... | ... | ... | ... |
| | | | |
| 1 | 1 | 5 | B |
| 1 | 2 | 7 | B |
| | | | |
| ... | ... | ... | ... |
| | | | |
+---------------------------------+
And we want to calculate the product of this matrixes: C = A x B
By definition: C_i_j = sum( A_i_k * B_k_j )
And here is a two-step MapReduce algorithm, for calculation of this product (I will provide a pseudocode):
First step:
function Map (input is a single row of the file from above):
i = row[0]
j = row[1]
value = row[2]
matrix = row[3]
if(matrix == 'A')
emit(i, {j, value, 'A'})
else
emit(j, {i, value, 'B'})
Complexity of this Map function is O(1)
function Reduce(Key, List of tuples from the Map function):
Matrix_A_tuples =
filter( List of tuples from the Map function, where matrix == 'A' )
Matrix_B_tuples =
filter( List of tuples from the Map function, where matrix == 'B' )
for each tuple_A from Matrix_A_tuples
i = tuple_A[0]
value_A = tuple_A[1]
for each tuple_B from Matrix_B_tuples
j = tuple_B[0]
value_B = tuple_B[1]
emit({i, j}, {value_A * value_b, 'C'})
Complexity of this Reduce function is O(N^2)
After the first step we will get something like the following file (which contains O(N^3) lines):
+---------------------------------+
| i | j | value | matrix |
+---------------------------------+
| 1 | 1 | 50 | C |
| 1 | 1 | 45 | C |
| | | | |
| ... | ... | ... | ... |
| | | | |
| 2 | 2 | 70 | C |
| 2 | 2 | 17 | C |
| | | | |
| ... | ... | ... | ... |
| | | | |
+---------------------------------+
So, all we have to do - just sum the values, from lines, which contains the same values i and j.
Second step:
function Map (input is a single row of the file, which produced in first step):
i = row[0]
j = row[1]
value = row[2]
emit({i, j}, value)
function Reduce(Key, List of values from the Map function)
i = Key[0]
j = Key[1]
result = 0;
for each Value from List of values from the Map function
result += Value
emit({i, j}, result)
After the second step we will get the file, which contains cells of the matrix C.
So the question is:
Taking into account, that there are multiple number of instances in MapReduce cluster - which is the most correct way to estimate complexity of the provided algorithm?
The first one, which comes to mind is such:
When we assume that number of instances in the MapReduce cluster is K.
And, because of the number of lines - from file, which produced after the first step is O(N^3) - the overall complexity can be estimated as O((N^3)/K).
But this estimation doesn't take into account many details: such as network bandwidth between instances of MapReduce cluster, ability to distribute data between distances - and perform most of the calculations locally etc.
So, I would like to know which is the best approach for estimation of efficiency of the provided MapReduce algorithm, and does it make sense to use Big-O notation to estimate efficiency of MapReduce algorithms at all?
as you said the Big-O estimates the computation complexity, and does not take into consideration the networking issues such(bandwidth, congestion, delay...)
If you want to calculate how much efficient the communication between instances, in this case you need other networking metrics...
However, I want to tell you something, if your file is not big enough, you will not see an improvement in term of execution speed. This is because the MapReduce works efficiently only with BIG data. Moreover, your code has two steps, that means two jobs. MapReduce, from one job to another, takes time to upload the file and start the job again. This can affect slightly the performance.
I think you can calculate the efficiently in term of speed and time as the MapReduce approach is for sure faster when it comes to big data. This is if we compared it to the sequential algorithms.
Moreover, efficiency can be with regards to the fault-tolerance. This is because MapReduce will manage to handle failures by itself. So, no need for the programmers to handle instance failure or networking failures..
Related
I have two tables. One has a list of items, and for each item, a number n.
item | n
--------
a | 1
b | 2
c | 3
The second one has a list of rows containing item, uid, and other rows.
item | uid | data
------------------
a | x | foo
a | x | baz
a | x | bar
a | z | arm
a | z | leg
b | x | eye
b | x | eye
b | x | eye
b | x | eye
b | z | tap
c | y | tip
c | z | top
I would like to sample, for each (item,uid) pair, n rows (arbitrary, it's better if this is uniformly random, but it doesn't have to be). In the example above, I want to keep maximum one row per user for item a, two rows per user for item b, and three rows per user to item c:
item | uid | data
------------------
a | x | baz
a | z | arm
b | x | eye
b | x | eye
b | z | tap
c | y | tip
c | z | top
ARRAY_AGG with LIMIT n doesn't work for two reasons: first, I suspect that given that n can be large (on the order of 100,000), this won't scale. The second, more fundamental problem is that n needs to be a constant.
Table sampling also doesn't seem to solve my problem, since it's per-table, and also only supports sampling a fixed percentage of rows, rather than a fixed number of rows.
Are there any other options?
Consider below solution
select * except(n)
from rows_list
join items_list
using(item)
where true
qualify row_number() over win <= n
window win as (partition by item, uid order by rand())
if applied to sample data in your question - output is
I'm measuring the performance of A and B programs. A is written in Golang, B is written in Python. The important point here is that I'm interested in how the performance value increases, not the absolute performance value of the two programs over time.
For example,
+------+-----+-----+
| time | A | B |
+------+-----+-----+
| 1 | 3 | 500 |
+------+-----+-----+
| 2 | 5 | 800 |
+------+-----+-----+
| 3 | 9 | 1300|
+------+-----+-----+
| 4 | 13 | 1800|
+------+-----+-----+
Where the values in columns A and B(A: 3, 5, 9, 13 / B: 500, 800, 1300, 1800) are the execution times of the program. This execution time can be seen as performance, and the difference between the absolute performance values of A and B is very large. Therefore, the slope comparison of the performance graphs of the two programs would be meaningless.(Python is very slow compared to Golang.)
I want to compare the performance of Program A written in Golang with Program B written in Python, and I'm looking for a calibration tool or formula based on benchmarks that calculates the execution time when Program A is written in Python.
Is there any way to solve this problem ?
If you are interested in the relative change, you should normalize the data for each programming language. In other words, divide the values for golang with 3 and for python, divide with value 500.
+------+-----+-----+
| time | A | B |
+------+-----+-----+
| 1 | 1 | 1 |
+------+-----+-----+
| 2 | 1.66| 1.6 |
+------+-----+-----+
| 3 | 3 | 2.6 |
+------+-----+-----+
| 4 |4.33 | 3.6 |
+------+-----+-----+
I have a drug analysis experiment that need to generate a value based on given drug database and set of 1000 random experiments.
The original database looks like this where the number in the columns represent the rank for the drug. This is a simplified version of actual database, the actual database will have more Drug and more Gene.
+-------+-------+-------+
| Genes | DrugA | DrugB |
+-------+-------+-------+
| A | 1 | 3 |
| B | 2 | 1 |
| C | 4 | 5 |
| D | 5 | 4 |
| E | 3 | 2 |
+-------+-------+-------+
A score is calculated based on user's input: A and C, using the following formula:
# Compute Function
# ['A','C'] as array input
computeFunction(array) {
# do some stuff with the array ...
}
The formula used will be same for any provided value.
For randomness test, each set of experiment requires the algorithm to provide randomized values of A and C, so both A and C can be having any number from 1 to 5
Now I have two methods of selecting value to generate the 1000 sets for P-Value calculation, but I would need someone to point out if there is one better than another, or if there is any method to compare these two methods.
Method 1
Generate 1000 randomized database based on given database input shown above, meaning all the table should contain different set of value pair.
Example for 1 database from 1000 randomized database:
+-------+-------+-------+
| Genes | DrugA | DrugB |
+-------+-------+-------+
| A | 2 | 3 |
| B | 4 | 4 |
| C | 3 | 2 |
| D | 1 | 5 |
| E | 5 | 1 |
+-------+-------+-------+
Next we perform computeFunction() with new A and C value.
Method 2
Pick any random gene from original database and use it as a newly randomized gene value.
For example, we pick the values from E and B as a new value for A and C.
From original database, E is 3, B is 2.
So, now A is 3, C is 2. Next we perform computeFunction() with new A and C value.
Summary
Since both methods produce completely randomized input, therefore it seems to me that it will produce similar 1000-value outcome. Is there any way I could prove they are similar?
Let a 3D grid, just like a checkerboard, with an extra dimension. Now let's say that I have a certain amount of cubes into that grid, each cube occupying 1x1x1 cells. Let's say that each of these cubes is an item.
What I would like to do is replace/combine these cubes into larger boxes occupying any number of cells on the X, Y and Z axes, so that the resulting number of boxes is as small as possible while preserving the overall "appearance".
It's probably unclear so I'll give a 2D example. Say I have a 2D grid containing several squares occupying 1x1 cells. A letter represents the cells occupied by a given item, each item having a different letter from the other ones. In the first example we have 10 different items, each of them occupying 1x1x1 cells.
+---+---+---+---+---+---+
| | | | | | |
+---+---+---+---+---+---+
| | A | B | C | D | |
+---+---+---+---+---+---+
| | E | F | G | H | |
+---+---+---+---+---+---+
| | | K | L | | |
+---+---+---+---+---+---+
| | | | | | |
+---+---+---+---+---+---+
That's my input data. I could now optimize it, i.e reduce the number of items while still occupying the same cells, by multiple possible ways, one of which could be :
+---+---+---+---+---+---+
| | | | | | |
+---+---+---+---+---+---+
| | A | B | B | C | |
+---+---+---+---+---+---+
| | A | B | B | C | |
+---+---+---+---+---+---+
| | | B | B | | |
+---+---+---+---+---+---+
| | | | | | |
+---+---+---+---+---+---+
Here, instead of 10 items, I only have 3 (i.e A, B and C). However it can be optimized even more :
+---+---+---+---+---+---+
| | | | | | |
+---+---+---+---+---+---+
| | A | A | A | A | |
+---+---+---+---+---+---+
| | A | A | A | A | |
+---+---+---+---+---+---+
| | | B | B | | |
+---+---+---+---+---+---+
| | | | | | |
+---+---+---+---+---+---+
Here I only have two items, A and B. This is as optimized as this can be.
What I am looking for is an algorithm capable of finding the best item sizes and arrangement, or at least a reasonably good one, so that I have as few items as possible while occupying the same cells, and in 3D !
Is there such an algorithm ? I'm sure there are some domains where that kind of algorithm would be useful, and I need it for a video game. Thanks !!
Perhaps a simpler algorithm is possible, but a set partition should suffice.
Min x1 + x2 + x3 + ... //where x1 is 1 if the 1th partition is chosen, 0 otherwise
such that x1 + + x3 = 1// if 1st and 3rd partition contain 1st item
x2 + x3 = 1//if 2nd and 3rd partition contain 2nd item and so on.
x1, x2, x3,... are binary
You have 1 constraint for each item. Each constraint stipulates that each item can be part of exactly one box. The objective minimizes the total number of boxes.
This is an NP Hard integer programming however.
The number of variables in this problem could be exponential. You need to have an efficient way of enumerating them -- that is figuring out when a contiguous box can be found that is capable of including all points in it. It is here that you have to take into account information such as whether the grid is 2d or 3d, how you define a contiguous "box", etc.
Such problems are usually solved by column-generation, where these columns of the integer program are dynamically generated on the fly.
If I understand David Eppstein's1 explanation (see section 3) then a solution can be found in a maximal independent set in the bipartite intersection graph of axis-aligned diagonals connecting one concave vertex to another. (This would be 2d. I'm not sure about 3d, although perhaps it involves evaluating hyperplanes instead of lines?)
In your example, there is only one such diagonal:
________
| |
|_x....x_|
|____|
The two xs represent connected concave vertices. The maximal independent set of edges here contains only one edge, splitting the polygon in two.
Here's another with only one axis-parallel edge connecting two concave vertices, x and x. This polygon, though, also has two concave vertices, a and b, that do not have an opposite, axis-parallel match. In that case, it seems to me, each concave vertex without a partner would split the polygon it's on in two (either vertically or horizontally):
____________
| |
| |x
| . |
| . |a
|___ . |
b| . |
| .___|
|________|x
results in 4 rectangles:
____________
| |
| |x
| . |
| ..|a
|___.......... |
b| . |
| .___|
|________|x
Here's one with two intersecting axis-parallel diagonals, each connecting two concave vertices, (x,x) and (y,y):
____________
| |
| |x_
| . |
| . |
|___ . . . .z. .|y
y| . |
| .____|
|________|x
In this case, as I understand, the intersection graph again contains only one independent set:
(y,z) (z,y) (x,z) (z,x)
yielding 4 rectangles as a solution.
Since I'm not completely sure how the "intersection graph" in the paper is defined, I would welcome any clarifying comments.
1. Graph-Theoretic Solutions to Computational Geometry Problems, David Eppstein (Submitted on 26 Aug 2009)
There is a N x M board we should paint. We can paint either an entire row or an entire column at once. Given an N x M matrix of colours of all board cells find the minimal number of painting operations to paint the board.
For example: we should paint a 3 x 3 board as follows (R - red, B - blue, G - green):
B, B, B
B, R, R
B, G, G
The minimal number of painting operations is 4:
Paint row 0 with Blue
Paint row 1 with Red
Paint row 2 with Green
Paint column 0 with Blue
How would you solve it ?
This looks like a fun problem. Let me take a shot at it with some pseudocode.
Function MinPaints(Matrix) Returns Integer
If the matrix is empty return 0
Find all rows and columns which have a single color
If there are none, return infinity, since there is no solution
Set the current minimum to infinity
For each row or column with single color:
Remove the row/column from the matrix
Call MinPaints with the new matrix
If the result is less than the current minimum, set the current minimum to the result
End loop
Return the current minimum + 1
End Function
I think that will solve your problem, but I didn't try any optimization or anything. This may not be fast enough though, I don't know. I doubt this problem is solvable in sub-exponential time.
Here is how this algorithm would solve the example:
BBB
BRR
BGG
|
+---BRR
| BGG
| |
| +---RR
| | GG
| | |
| | +---GG
| | | |
| | | +---[]
| | | | |
| | | | Solvable in 0
| | | |
| | | Solvable in 1
| | |
| | +---RR
| | | |
| | | +---[]
| | | | |
| | | | Solvable in 0
| | | |
| | | Solvable in 1
| | |
| | Solvable in 2
| |
| Solvable in 3
| BB
+---Another branch with RR ...
| GG
Solvable in 4
For starters, you can try an informed exhaustive search.
Let your states graph be: G=(V,E) where V = {all possible boards} and E = {(u,v) | you can move from board u to v within a single operation}.
Note that you do not need to generate the graph in advance - you can generate it on the fly, using a successors(board) function, that return all the successors of the given board.
You will also need h:V->R - an admissible heuristic function that evaluates the board1.
Now, you can run A*, or bi-directional BFS search [or combination of both], your source will be a white board, and your target is the requested board. Because we use admissible heuristic function - A* is both complete (always finds a solution if one exists) and optimal (finds the shortest solution), it will find the best solution. [same goes for bi-directional BFS].
drawbacks:
Though the algorithm is informed, it will have exponential behavior.
But if it is an interview question, I believe a non-efficient
solution is better then no solution.
Though complete and optimal - if there is no solution - the algorithm may be stuck in an infinite loop, or a very long loop at the very least until it realizes it has exuahsted all possibilities.
(1) example for admissible heuristic is h(board) = #(miscolored_squares)/max{m,n}