I would like to count the same log messages in Kibana. With the Size set to 200, it turns out that there are two results that happened twice
But, if I lower the Size to 5, I don't see those two:
It should show me top 5 rows, ordered by count. I expected something like this:
| LogMessage | Count |
|------------|-------|
| xx | 2 |
| yy | 2 |
| zz | 1 |
| qq | 1 |
| ww | 1 |
What am I missing?
The issue is the little warning about Analyzed Field. You should use a keyword field.
With analyzed fields, the analyzer breaks down the original string during indexing into sub-strings to facilitate search use cases (handling things like word boundaries, punctuation, case insensitivity, declination, etc)
A keyword field is just a simple string.
What's probably happening is that you have data like
| LogMessage | Count |
|------------|-------|
| a | 1 |
| b | 1 |
| c x | 1 |
| d x | 1 |
With an analyzed field, if you have a terms agg of size 2 you might (depending on the sort order) get a and b
With a larger terms agg, the top sub-string will be x
This is a simplified example, but I hope it gets the issue across.
The Terms Aggregation docs have a good section about how to avoid/solve this issue.
Related
I have a very simple graph with 5 nodes (named n1 - n5), 1 node type (:Node) and 2 relationship types (:r1, :r2). The nodes and relationships are arranged as follows (apologies for the ascii art):
(n1)-[:r1]->(n2)-[:r1]->(n3)
(n1)-[:r2]->(n4)-[:r2]->(n3)
(n1)-[:r1]->(n5)-[:r2]->(n3)
I have a query using a variable length path. I expected to be able restrict the paths returned by describing a specific pattern in the WHERE clause:
MATCH p = (n:Node {name: 'n1'})-[*..2]->()
WHERE (n)-[:r1]->()-[:r1]->()
RETURN p
The problem is that the response returns all possible paths. My question; is it possible to filter the returned paths when specifying a variable length path in a query?
If all relationships or nodes have to adhere to the same predicate, this is easy. You'll need a variable for the path, and you'll need to use all() (or none()) in your WHERE clause to apply the predicate for all relationships or nodes in your path:
MATCH p = (n:Node {name: 'n1'})-[*..2]->()
WHERE all(rel in relationships(p) WHERE type(rel) = 'r1')
RETURN p
That said, when all you want is for all relationships in the var-length path to be of the same type (or types, if you want multiple), that's best done in the pattern itself:
MATCH p = (n:Node {name: 'n1'})-[:r1*..2]->()
RETURN p
For more complicated cases, such as multiple relationship types (where the order of those types matters in the path), or repeating sequences of types or node labels in the path, then alternate approaches are needed. APOC path expanders may help.
EDIT
You mentioned in the comments that your case deals with sequences of relationships of varying lengths. While the APOC path expanders may help, it there are a few restrictions:
The path expanders currently operate on node labels and relationship types, but not properties, so if your expansions rely on predicates on properties, the path expanders won't be able to handle that for you during expansion, that would have to be done by filtering the path expander results after.
There are limits to the relationship sequence support for path expanders. We can define sequences of any length, and can accept multiple relationship types at each step in the sequence, but we don't currently support diverging sequences ((r1 then r2 then r3) or (r2 then r5 then r6)).
If we wanted to do a 3-step sequence of r1 (incoming), r2 (outgoing), then r3 or r4 (with r3 in either direction and r4 outgoing), repeating the sequence up to 3 times we could do so like this:
MATCH (n:Node {name: 'n1'})
CALL apoc.path.expandConfig(n, {relationshipFilter:'<r1, r2>, r3 | r4>', minLevel:1, maxLevel:9) YIELD path
RETURN path
Note that we can provide differing directions per relationship in the filter, or leave off the arrow entirely if we don't care about the direction.
Label filtering is more complex, but I didn't see any need for that present in the examples so far.
Your query return all paths because your WHERE clause (Filter operator) is applied before the VarLengthExpand operator:
+-----------------------+----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
| Operator | Estimated Rows | Rows | DB Hits | Page Cache Hits | Page Cache Misses | Page Cache Hit Ratio | Variables | Other |
+-----------------------+----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
| +ProduceResults | 0 | 4 | 0 | 0 | 0 | 0.0000 | anon[32], anon[41], n, p | |
| | +----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
| +Projection | 0 | 4 | 0 | 0 | 0 | 0.0000 | p -- anon[32], anon[41], n | {p : PathExpression(NodePathStep(Variable(n),MultiRelationshipPathStep(Variable(),OUTGOING,NilPathStep)))} |
| | +----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
| +VarLengthExpand(All) | 0 | 4 | 7 | 0 | 0 | 0.0000 | anon[32], anon[41] -- n | (n)-[:*..2]->() |
| | +----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
| +Filter | 0 | 1 | 6 | 0 | 0 | 0.0000 | n | n.name = { AUTOSTRING0}; GetDegree(Variable(n),Some(RelTypeName(KNOWS)),OUTGOING) > 0 |
| | +----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
| +NodeByLabelScan | 4 | 4 | 5 | 0 | 0 | 0.0000 | n | :Crew |
+-----------------------+----------------+------+---------+-----------------+-------------------+----------------------+----------------------------+------------------------------------------------------------------------------------------------------------+
This should get you going:
MATCH p = (n:Node {name: 'n1'})-[*..2]->()
WITH n, relationships(p)[0] as rel0, relationships(p)[1] as rel1, p
MATCH (n)-[rel0:r1]->()-[rel1:r1]->()
RETURN p
I have a drug analysis experiment that need to generate a value based on given drug database and set of 1000 random experiments.
The original database looks like this where the number in the columns represent the rank for the drug. This is a simplified version of actual database, the actual database will have more Drug and more Gene.
+-------+-------+-------+
| Genes | DrugA | DrugB |
+-------+-------+-------+
| A | 1 | 3 |
| B | 2 | 1 |
| C | 4 | 5 |
| D | 5 | 4 |
| E | 3 | 2 |
+-------+-------+-------+
A score is calculated based on user's input: A and C, using the following formula:
# Compute Function
# ['A','C'] as array input
computeFunction(array) {
# do some stuff with the array ...
}
The formula used will be same for any provided value.
For randomness test, each set of experiment requires the algorithm to provide randomized values of A and C, so both A and C can be having any number from 1 to 5
Now I have two methods of selecting value to generate the 1000 sets for P-Value calculation, but I would need someone to point out if there is one better than another, or if there is any method to compare these two methods.
Method 1
Generate 1000 randomized database based on given database input shown above, meaning all the table should contain different set of value pair.
Example for 1 database from 1000 randomized database:
+-------+-------+-------+
| Genes | DrugA | DrugB |
+-------+-------+-------+
| A | 2 | 3 |
| B | 4 | 4 |
| C | 3 | 2 |
| D | 1 | 5 |
| E | 5 | 1 |
+-------+-------+-------+
Next we perform computeFunction() with new A and C value.
Method 2
Pick any random gene from original database and use it as a newly randomized gene value.
For example, we pick the values from E and B as a new value for A and C.
From original database, E is 3, B is 2.
So, now A is 3, C is 2. Next we perform computeFunction() with new A and C value.
Summary
Since both methods produce completely randomized input, therefore it seems to me that it will produce similar 1000-value outcome. Is there any way I could prove they are similar?
I have a table abcd in Oracle DB
+-------------+----------+
| abcd.speed | abcd.ab |
+-------------+----------+
| 4.0 | 2 |
| 4.0 | 2 |
| 7.0 | 2 |
| 7.0 | 2 |
| 8.0 | 1 |
+-------------+----------+
And I'm using a query like this:
select min(speed) keep (dense_rank last order by abcd.ab NULLS FIRST) MOD from abcd;
I'm trying to convert the code to Hive, but it looks like keep is not available in Hive.
Could you suggest an equivalent statement?
select -max(struct(ab,-speed)).col2 as mod
from abcd
;
+------+
| mod |
+------+
| 4.0 |
+------+
Let start by explaining min(speed) keep (dense_rank last order by abcd.ab NULLS FIRST):
Find the row(s) with the max value of ab.
For this/those row(s), find the min value of speed.
We are using 2 tricks here.
The 1st is based on the ability to get the max value of a struct.
max(struct(c1,c2,c3,...)) returns the same result as if you have sorted the structs by c1, then by c2, then by c3 etc. and then chose the last element.
The 2nd trick is to use -speed (which is the same of -1*speed).
Finding the max of -speed and then taking the minus of that value (which gives us speed), is the same of finding the min of speed.
If we would have ordered the structs, it would have looked like this (since 2 is bigger than 1 and -4 is bigger than -7):
+----+-------+
| ab | speed |
+----+-------+
| 1 | -8.0 |
| 2 | -7.0 |
| 2 | -7.0 |
| 2 | -4.0 |
| 2 | -4.0 |
+----+-------+
The last struct in this case in struct(2,-4.0), therefore this is the result of the max function.
The fields names for a struct are col1, col2, col3 etc., so
struct(2,-4.0).col2 is -4.0. and preceding it with minus (which is the same as multiple it by -1) as in -struct(2,-4.0).col2 is 4.0.
Using Derby, is it possible to offset by a value from the query rather than an integer literal?
When I run this query, it complains about the value I've given to the offset clause.
select
PRIZE."NAME" as "Prize Name",
PRIZE."POSITION" as "Position",
(select
PARTICIPANT."NAME"
from PARTICIPANT
order by POINTS desc
offset PRIZE."POSITION" rows fetch next 1 row only <-- notice I'm trying to pass in a value to offset by
) as "Participant"
from PRIZE
With the expectation that the results would look like this:
| Prize Name | Position | Participant |
|--------------|----------|---------------|
| Gold medal | 1 | Mari Loudi |
| Silver medal | 2 | Keesha Vacc |
| Bronze medal | 3 | Melba Hammit |
| Hundredth | 100 | James Thornby |
The documentation suggests that it's possible to pass in a value from java code, but I'm trying to use a value from the query itself.
By the way, this is just an example schema to illustrate the point.
I know there are other ways to achieve the ranking, but I'm specifically interested if there's a way to pass values to the offset clause.
Assume, that we have large file, which contains descriptions of the cells of two matrices (A and B):
+---------------------------------+
| i | j | value | matrix |
+---------------------------------+
| 1 | 1 | 10 | A |
| 1 | 2 | 20 | A |
| | | | |
| ... | ... | ... | ... |
| | | | |
| 1 | 1 | 5 | B |
| 1 | 2 | 7 | B |
| | | | |
| ... | ... | ... | ... |
| | | | |
+---------------------------------+
And we want to calculate the product of this matrixes: C = A x B
By definition: C_i_j = sum( A_i_k * B_k_j )
And here is a two-step MapReduce algorithm, for calculation of this product (I will provide a pseudocode):
First step:
function Map (input is a single row of the file from above):
i = row[0]
j = row[1]
value = row[2]
matrix = row[3]
if(matrix == 'A')
emit(i, {j, value, 'A'})
else
emit(j, {i, value, 'B'})
Complexity of this Map function is O(1)
function Reduce(Key, List of tuples from the Map function):
Matrix_A_tuples =
filter( List of tuples from the Map function, where matrix == 'A' )
Matrix_B_tuples =
filter( List of tuples from the Map function, where matrix == 'B' )
for each tuple_A from Matrix_A_tuples
i = tuple_A[0]
value_A = tuple_A[1]
for each tuple_B from Matrix_B_tuples
j = tuple_B[0]
value_B = tuple_B[1]
emit({i, j}, {value_A * value_b, 'C'})
Complexity of this Reduce function is O(N^2)
After the first step we will get something like the following file (which contains O(N^3) lines):
+---------------------------------+
| i | j | value | matrix |
+---------------------------------+
| 1 | 1 | 50 | C |
| 1 | 1 | 45 | C |
| | | | |
| ... | ... | ... | ... |
| | | | |
| 2 | 2 | 70 | C |
| 2 | 2 | 17 | C |
| | | | |
| ... | ... | ... | ... |
| | | | |
+---------------------------------+
So, all we have to do - just sum the values, from lines, which contains the same values i and j.
Second step:
function Map (input is a single row of the file, which produced in first step):
i = row[0]
j = row[1]
value = row[2]
emit({i, j}, value)
function Reduce(Key, List of values from the Map function)
i = Key[0]
j = Key[1]
result = 0;
for each Value from List of values from the Map function
result += Value
emit({i, j}, result)
After the second step we will get the file, which contains cells of the matrix C.
So the question is:
Taking into account, that there are multiple number of instances in MapReduce cluster - which is the most correct way to estimate complexity of the provided algorithm?
The first one, which comes to mind is such:
When we assume that number of instances in the MapReduce cluster is K.
And, because of the number of lines - from file, which produced after the first step is O(N^3) - the overall complexity can be estimated as O((N^3)/K).
But this estimation doesn't take into account many details: such as network bandwidth between instances of MapReduce cluster, ability to distribute data between distances - and perform most of the calculations locally etc.
So, I would like to know which is the best approach for estimation of efficiency of the provided MapReduce algorithm, and does it make sense to use Big-O notation to estimate efficiency of MapReduce algorithms at all?
as you said the Big-O estimates the computation complexity, and does not take into consideration the networking issues such(bandwidth, congestion, delay...)
If you want to calculate how much efficient the communication between instances, in this case you need other networking metrics...
However, I want to tell you something, if your file is not big enough, you will not see an improvement in term of execution speed. This is because the MapReduce works efficiently only with BIG data. Moreover, your code has two steps, that means two jobs. MapReduce, from one job to another, takes time to upload the file and start the job again. This can affect slightly the performance.
I think you can calculate the efficiently in term of speed and time as the MapReduce approach is for sure faster when it comes to big data. This is if we compared it to the sequential algorithms.
Moreover, efficiency can be with regards to the fault-tolerance. This is because MapReduce will manage to handle failures by itself. So, no need for the programmers to handle instance failure or networking failures..