Enumerating Cartesian product while minimizing repetition - algorithm

Given two sets, e.g.:
{A B C}, {1 2 3 4 5 6}
I want to generate the Cartesian product in an order that puts as much space as possible between equal elements. For example, [A1, A2, A3, A4, A5, A6, B1…] is no good because all the As are next to each other. An acceptable solution would be going "down the diagonals" and then every time it wraps offsetting by one, e.g.:
[A1, B2, C3, A4, B5, C6, A2, B3, C4, A5, B6, C1, A3…]
Expressed visually:
| | A | B | C | A | B | C | A | B | C | A | B | C | A | B | C | A | B | C |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | | | | | | | | | | | | | | | | | |
| 2 | | 2 | | | | | | | | | | | | | | | | |
| 3 | | | 3 | | | | | | | | | | | | | | | |
| 4 | | | | 4 | | | | | | | | | | | | | | |
| 5 | | | | | 5 | | | | | | | | | | | | | |
| 6 | | | | | | 6 | | | | | | | | | | | | |
| 1 | | | | | | | | | | | | | | | | | | |
| 2 | | | | | | | 7 | | | | | | | | | | | |
| 3 | | | | | | | | 8 | | | | | | | | | | |
| 4 | | | | | | | | | 9 | | | | | | | | | |
| 5 | | | | | | | | | | 10| | | | | | | | |
| 6 | | | | | | | | | | | 11| | | | | | | |
| 1 | | | | | | | | | | | | 12| | | | | | |
| 2 | | | | | | | | | | | | | | | | | | |
| 3 | | | | | | | | | | | | | 13| | | | | |
| 4 | | | | | | | | | | | | | | 14| | | | |
| 5 | | | | | | | | | | | | | | | 15| | | |
| 6 | | | | | | | | | | | | | | | | 16| | |
| 1 | | | | | | | | | | | | | | | | | 17| |
| 2 | | | | | | | | | | | | | | | | | | 18|
or, equivalently but without repeating the rows/columns:
| | A | B | C |
|---|----|----|----|
| 1 | 1 | 17 | 15 |
| 2 | 4 | 2 | 18 |
| 3 | 7 | 5 | 3 |
| 4 | 10 | 8 | 6 |
| 5 | 13 | 11 | 9 |
| 6 | 16 | 14 | 12 |
I imagine there are other solutions too, but that's the one I found easiest to think about. But I've been banging my head against the wall trying to figure out how to express it generically—it's a convenient thing that the cardinality of the two sets are multiples of each other, but I want the algorithm to do The Right Thing for sets of, say, size 5 and 7. Or size 12 and 69 (that's a real example!).
Are there any established algorithms for this? I keep getting distracted thinking of how rational numbers are mapped onto the set of natural numbers (to prove that they're countable), but the path it takes through ℕ×ℕ doesn't work for this case.
It so happens the application is being written in Ruby, but I don't care about the language. Pseudocode, Ruby, Python, Java, Clojure, Javascript, CL, a paragraph in English—choose your favorite.
Proof-of-concept solution in Python (soon to be ported to Ruby and hooked up with Rails):
import sys
letters = sys.argv[1]
MAX_NUM = 6
letter_pos = 0
for i in xrange(MAX_NUM):
for j in xrange(len(letters)):
num = ((i + j) % MAX_NUM) + 1
symbol = letters[letter_pos % len(letters)]
print "[%s %s]"%(symbol, num)
letter_pos += 1

String letters = "ABC";
int MAX_NUM = 6;
int letterPos = 0;
for (int i=0; i < MAX_NUM; ++i) {
for (int j=0; j < MAX_NUM; ++j) {
int num = ((i + j) % MAX_NUM) + 1;
char symbol = letters.charAt(letterPos % letters.length);
String output = symbol + "" + num;
++letterPos;
}
}

What about using something fractal/recursive? This implementation divides a rectangular range into four quadrants then yields points from each quadrant. This means that neighboring points in the sequence differ at least by quadrant.
#python3
import sys
import itertools
def interleave(*iters):
for elements in itertools.zip_longest(*iters):
for element in elements:
if element != None:
yield element
def scramblerange(begin, end):
width = end - begin
if width == 1:
yield begin
else:
first = scramblerange(begin, int(begin + width/2))
second = scramblerange(int(begin + width/2), end)
yield from interleave(first, second)
def scramblerectrange(top=0, left=0, bottom=1, right=1, width=None, height=None):
if width != None and height != None:
yield from scramblerectrange(bottom=height, right=width)
raise StopIteration
if right - left == 1:
if bottom - top == 1:
yield (left, top)
else:
for y in scramblerange(top, bottom):
yield (left, y)
else:
if bottom - top == 1:
for x in scramblerange(left, right):
yield (x, top)
else:
halfx = int(left + (right - left)/2)
halfy = int(top + (bottom - top)/2)
quadrants = [
scramblerectrange(top=top, left=left, bottom=halfy, right=halfx),
reversed(list(scramblerectrange(top=top, left=halfx, bottom=halfy, right=right))),
scramblerectrange(top=halfy, left=left, bottom=bottom, right=halfx),
reversed(list(scramblerectrange(top=halfy, left=halfx, bottom=bottom, right=right)))
]
yield from interleave(*quadrants)
if __name__ == '__main__':
letters = 'abcdefghijklmnopqrstuvwxyz'
output = []
indices = dict()
for i, pt in enumerate(scramblerectrange(width=11, height=5)):
indices[pt] = i
x, y = pt
output.append(letters[x] + str(y))
table = [[indices[x,y] for x in range(11)] for y in range(5)]
print(', '.join(output))
print()
pad = lambda i: ' ' * (2 - len(str(i))) + str(i)
header = ' |' + ' '.join(map(pad, letters[:11]))
print(header)
print('-' * len(header))
for y, row in enumerate(table):
print(pad(y)+'|', ' '.join(map(pad, row)))
Outputs:
a0, i1, a2, i3, e0, h1, e2, g4, a1, i0, a3, k3, e1,
h0, d4, g3, b0, j1, b2, i4, d0, g1, d2, h4, b1, j0,
b3, k4, d1, g0, d3, f4, c0, k1, c2, i2, c1, f1, a4,
h2, k0, e4, j3, f0, b4, h3, c4, j2, e3, g2, c3, j4,
f3, k2, f2
| a b c d e f g h i j k
-----------------------------------
0| 0 16 32 20 4 43 29 13 9 25 40
1| 8 24 36 28 12 37 21 5 1 17 33
2| 2 18 34 22 6 54 49 39 35 47 53
3| 10 26 50 30 48 52 15 45 3 42 11
4| 38 44 46 14 41 31 7 23 19 51 27

If your sets X and Y are sizes m and n, and Xi is the index of the element from X that's in the ith pair in your Cartesian product (and similar for Y), then
Xi = i mod n;
Yi = (i mod n + i div n) mod m;
You could get your diagonals a little more spread out by filling out your matrix like this:
for (int i = 0; i < m*n; i++) {
int xi = i % n;
int yi = i % m;
while (matrix[yi][xi] != 0) {
yi = (yi+1) % m;
}
matrix[yi][xi] = i+1;
}

Related

Pivot Table in Hive and Create Multiple Columns for Unique Combinations

I want to pivot the following table
| ID | Code | date | qty |
| 1 | A | 1/1/19 | 11 |
| 1 | A | 2/1/19 | 12 |
| 2 | B | 1/1/19 | 13 |
| 2 | B | 2/1/19 | 14 |
| 3 | C | 1/1/19 | 15 |
| 3 | C | 3/1/19 | 16 |
into
| ID | Code | mth_1(1/1/19) | mth_2(2/1/19) | mth_3(3/1/19) |
| 1 | A | 11 | 12 | 0 |
| 2 | B | 13 | 14 | 0 |
| 3 | C | 15 | 0 | 16 |
I am new to hive, i am not sure how to implement it.
NOTE: I don't want to do mapping because my month values change over time.

Elixir: to assign variable in for generator(variable scope?)

I'm solving, find the largest prime factor of the number, Project Euler problem3.
Following Elixir code throw warnings, and do not evaluate in if block(assigning) I think:
num = 13195
range = num
|> :math.sqrt
|> Float.floor
|> round
for dv <- 2..range do
if rem(num, dv) == 0 and div(num, dv) != 1 do
num = div(num, dv)
end
end
num
|> IO.puts
Warnings are:
$ elixir 3.exs
warning: variable "num" is unused
3.exs:10
warning: the result of the expression is ignored (suppress the warning by assigning the expression to the _ variable)
3.exs:10
13195
$ elixir -v
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Elixir 1.5.3
How can I update(assign) the num?
(following Python and Javascript codes are work for same the problem):
# 3.py
from math import ceil, sqrt
num = 600851475143
for div in range(2, ceil(sqrt(num)) + 1):
if num%div == 0 and num/div != 1:
num /= div
assert int(num) == 6857
// 3.js
var num = 600851475143;
var range = Array.from({length: Math.trunc(Math.sqrt(num))}, (x, i) => i + 2)
for (const div of range) {
if (num%div === 0 && num/div != 1) {
num /= div;
}
}
var assert = require('assert');
assert(num === 6857)
You are actually creating a new variable and shadowing the one from outer scope
You can rewrite it like this
num = 13195
range =
num
|> :math.sqrt()
|> Float.floor()
|> round
num =
2..range
|> Enum.reduce(num, fn elem, acc ->
if rem(acc, elem) == 0 and div(acc, elem) != 1 do
div(acc, elem)
else
acc
end
end)
IO.puts num
More on shadowing:
+------------------------------------------------------------+
| Top level |
| |
| +------------------------+ +------------------------+ |
| | Module | | Module | |
| | | | | |
| | +--------------------+ | | +--------------------+ | |
| | | Function clause | | | | Function clause | | |
| | | | | | | | | |
| | | +----------------+ | | | | +----------------+ | | |
| | | | Comprehension | | | | | | Comprehension | | | |
| | | +----------------+ | | | | +----------------+ | | |
| | | +----------------+ | | ... | | +----------------+ | | |
| | | | Anon. function | | | | | | Anon. function | | | |
| | | +----------------+ | | | | +----------------+ | | |
| | | +----------------+ | | | | +----------------+ | | |
| | | | Try block | | | | | | Try block | | | |
| | | +----------------+ | | | | +----------------+ | | |
| | +--------------------+ | | +--------------------+ | |
| +------------------------+ +------------------------+ |
| |
+------------------------------------------------------------+
Any variable in a nested scope whose name coincides with a variable from the surrounding scope will shadow that outer variable. In other words, the variable inside the nested scope temporarily hides the variable from the surrounding scope, but does not affect it in any way.
source

Turing Machine algorithms

I'm trying to determine how a Turing Machine (consisting of only 0's and 1's, no blanks) could recognize a sequence of 8 1's. Every algorithm I've found has a TM searching for an indeterminate number of 1's or 0's, not a specific number.
Essentially, if you have this tape:
1 1 1 1 1 1 1 1 0 0 0 1 0 0 0 1
How can you recognize that the 8 1's represent addition, and you want to add 0 0 0 1 and 0 0 0 1?
I take it that 11111111 is like an opcode and 0001, 0001 are the operands for that opcode. At least, that's the only interpretation I am seeing.
A TM can look for a fixed, finite number of symbols by using a similar fixed, finite number of states, the sole purpose of each one being to recognize that another of the expected symbols has been seen. For instance, here's a four-tape TM that recognizes addition and does the binary addition:
|----|----|----|----|----||----|-----|-----|-----|-----|----|----|----|----|
| Q | T1 | T2 | T3 | T4 || Q' | T1' | T2' | T3' | T4' | D1 | D2 | D3 | D4 |
|----|----|----|----|----||----|-----|-----|-----|-----|----|----|----|----|
// read the opcode /////////////////////////////////////////////////////////
| qA | 1 | x | y | z || qB | 1 | x | y | z | R | S | S | S |
| qB | 1 | x | y | z || qC | 1 | x | y | z | R | S | S | S |
| qC | 1 | x | y | z || qD | 1 | x | y | z | R | S | S | S |
| qD | 1 | x | y | z || qE | 1 | x | y | z | R | S | S | S |
| qE | 1 | x | y | z || qF | 1 | x | y | z | R | S | S | S |
| qF | 1 | x | y | z || qG | 1 | x | y | z | R | S | S | S |
| qG | 1 | x | y | z || qH | 1 | x | y | z | R | S | S | S |
| qH | 1 | x | y | z || qI | 1 | x | y | z | R | S | S | S |
// read the first addend ///////////////////////////////////////////////////
| qI | w | x | y | z || qJ | w | w | y | z | R | R | S | S |
| qJ | w | x | y | z || qK | w | w | y | z | R | R | S | S |
| qK | w | x | y | z || qL | w | w | y | z | R | R | S | S |
| qL | w | x | y | z || qM | w | w | y | z | R | R | S | S |
// read the second addend //////////////////////////////////////////////////
| qM | w | x | y | z || qN | w | x | w | z | R | S | R | S |
| qN | w | x | y | z || qO | w | x | w | z | R | S | R | S |
| qO | w | x | y | z || qP | w | x | w | z | R | S | R | S |
| qP | w | x | y | z || qQ | w | x | w | z | R | S | R | S |
// prepare the output tape /////////////////////////////////////////////////
| qQ | w | x | y | z || qR | w | x | y | z | S | S | S | R |
| qR | w | x | y | z || qS | w | x | y | z | S | S | S | R |
| qS | w | x | y | z || qT | w | x | y | z | S | S | S | R |
| qT | w | x | y | z || qU | w | x | y | z | S | S | S | R |
// handle addition when no carry ///////////////////////////////////////////
| qU | w | 0 | 0 | z || qU | w | 0 | 0 | 0 | S | L | L | L |
| qU | w | 0 | 1 | z || qU | w | 0 | 1 | 1 | S | L | L | L |
| qU | w | 1 | 0 | z || qU | w | 1 | 0 | 1 | S | L | L | L |
| qU | w | 1 | 1 | z || qV | w | 1 | 1 | 0 | S | L | L | L |
| qU | w | B | B | B || hA | w | B | B | B | S | R | R | R |
// handle addition when carry //////////////////////////////////////////////
| qV | w | 0 | 0 | z || qU | w | 0 | 0 | 1 | S | L | L | L |
| qV | w | 0 | 1 | z || qV | w | 0 | 1 | 0 | S | L | L | L |
| qV | w | 1 | 0 | z || qV | w | 1 | 0 | 0 | S | L | L | L |
| qV | w | 1 | 1 | z || qV | w | 1 | 1 | 1 | S | L | L | L |
| qV | w | B | B | B || hA | w | B | B | B | S | R | R | R |
|----|----|----|----|----||----|-----|-----|-----|-----|----|----|----|----|
Legend:
Q: current state
T1: current tape symbol, input tape
T2: current tape symbol, scratch tape #1
T3: current tape symbol, scratch tape #2
T4: current tape symbol, output tape (not used)
Q': state to transition into
T1': symbol to write to input tape (not used)
T2': symbol to write to scratch tape #1
T3': symbol to write to scratch tape #2
T4': symbol to write to output tape
D1: direction to move input tape head
D2: direction to move scratch tape #1 head
D3: direction to move scratch tape #2 head
D4: direction to move output tape head
Conventions:
w, x, y and z are variables and represent either 0 or 1. A transition using all four of these can be thought of as a shorthand notation for writing sixteen (2^4) concrete transitions.
directions are L=left, S=same, R=right.
B is a blank symbol; it can be dispensed with if you add more states to assist U and V in the addition.

Neo4j query performance with memory tuning

I have a graph (User-[Likes]->Item) with millions nodes and billions nodes (roughly 50G in disk) built on a powerful machine with 256G RAM and 40 cores. Currently, I'm computing the allshortestpath() between two items.
To improve the cypher query performance, I set dbms.pagecache.memory=100g and wrapper.java.additional=-Xmx32g, with the hope that the whole neo4j can be loaded into meomory. However, when I execute the shortestpath query, the CPU usage is 1625% while MEMORY usage is only 5.7%, and I didn't see performance improvements on the cypher query. Am I missing something in the setting? Or can I setup something to run the query faster? I have read the Performance Tuning guide in the developer manual but didn't find solution.
EDIT1:
The cypher query is to count the number of unique users that like both two items. The full pattern would be (Brand)-[:Has]->(Item)<-[:LIKES]-(User)-[:LIKES]->(Item)<-[:HAS]-(Brand)
profile
MATCH p = allShortestPaths((p1:Brand {FID:'001'})-[*..4]-(p2:Brand {FID:'002'}))
with [r in RELS(p)|type(r)] as relationshipPath,
[n in nodes(p)|id(n)][2] as user, p1, p2
return p1.FID, p2.FID, count(distinct user);
EDIT2:
Below is a sampler query plan. It now seems that I'm not using shortestsPath efficiently (380,556,69 db hits). I use shortestsPath to get the common user node between start/end nodes, and then use count(distinct) to get the unique user. Is it possible to tell cypher to eliminate paths which contain the node that have been visited before?
Can you try to run this instead:
MATCH (p1:Brand {FID:'001'}),(p2:Brand {FID:'002'})
MATCH (u:User)
WHERE (p1)-[:Has]->()<-[:LIKES]-(u) AND
(p2)-[:Has]->()<-[:LIKES]-(u)
RETURN p1,p2,count(u);
This starts at the user and checks against both brands, the explain plan looks much better
+----------------------+----------------+------------------------------------------+---------------------------+
| Operator | Estimated Rows | Variables | Other |
+----------------------+----------------+------------------------------------------+---------------------------+
| +ProduceResults | 0 | count(u), p1, p2 | p1, p2, count(u) |
| | +----------------+------------------------------------------+---------------------------+
| +EagerAggregation | 0 | count(u) -- p1, p2 | p1, p2 |
| | +----------------+------------------------------------------+---------------------------+
| +SemiApply | 0 | p2 -- p1, u | |
| |\ +----------------+------------------------------------------+---------------------------+
| | +Expand(Into) | 0 | anon[78] -- anon[87], anon[89], p1, u | (p1)-[:Has]->() |
| | | +----------------+------------------------------------------+---------------------------+
| | +Expand(All) | 0 | anon[87], anon[89] -- p1, u | (u)-[:LIKES]->() |
| | | +----------------+------------------------------------------+---------------------------+
| | +Argument | 1 | p1, u | |
| | +----------------+------------------------------------------+---------------------------+
| +SemiApply | 0 | p1 -- p2, u | |
| |\ +----------------+------------------------------------------+---------------------------+
| | +Expand(Into) | 0 | anon[119] -- anon[128], anon[130], p2, u | (p2)-[:Has]->() |
| | | +----------------+------------------------------------------+---------------------------+
| | +Expand(All) | 0 | anon[128], anon[130] -- p2, u | (u)-[:LIKES]->() |
| | | +----------------+------------------------------------------+---------------------------+
| | +Argument | 1 | p2, u | |
| | +----------------+------------------------------------------+---------------------------+
| +CartesianProduct | 0 | u -- p1, p2 | |
| |\ +----------------+------------------------------------------+---------------------------+
| | +CartesianProduct | 0 | p2 -- p1 | |
| | |\ +----------------+------------------------------------------+---------------------------+
| | | +Filter | 0 | p1 | p1.FID == { AUTOSTRING0} |
| | | | +----------------+------------------------------------------+---------------------------+
| | | +NodeByLabelScan | 0 | p1 | :Brand |
| | | +----------------+------------------------------------------+---------------------------+
| | +Filter | 0 | p2 | p2.FID == { AUTOSTRING1} |
| | | +----------------+------------------------------------------+---------------------------+
| | +NodeByLabelScan | 0 | p2 | :Brand |
| | +----------------+------------------------------------------+---------------------------+
| +NodeByLabelScan | 0 | u | :User |
+----------------------+----------------+------------------------------------------+---------------------------+

Is it normal query performance?

I have next graph model:
(:PaveView {Number:int, Page:string}), (:Page {Name:string})
(:PageView)-[:At]->(:Page)
(:PageView)-[:Next]->(:PageView)
Schema:
Indexes
ON :Page(Name) ONLINE (for uniqueness constraint)
ON :PageView(Page) ONLINE
ON :PageView(Revision) ONLINE (for uniqueness constraint)
Constraints
ON (pageview:PageView) ASSERT pageview.Number IS UNIQUE
ON (page:Page) ASSERT page.Name IS UNIQUE
I want to do something similar to this post
I have tried to find popular paths without loops of this structure:
(:PageView)-[:Next*2]->(:PageView)
That my tries:
1. Nicole White's method from post
MATCH p = (:PageView)-[:Next*2]->(:PageView)
WITH p, EXTRACT(v IN NODES(p) | v.Page) AS pages
UNWIND pages AS views
WITH p, COUNT(DISTINCT views) AS distinct_views
WHERE distinct_views = LENGTH(NODES(p))
RETURN EXTRACT(v in NODES(p) | v.Page), count(p)
ORDER BY count(p) DESC
LIMIT 10;
profile output:
10 rows
177270 ms
Compiler CYPHER 2.2-rule
ColumnFilter(0)
|
+Extract(0)
|
+ColumnFilter(1)
|
+Top
|
+EagerAggregation(0)
|
+Extract(1)
|
+ColumnFilter(2)
|
+Filter(0)
|
+Extract(2)
|
+ColumnFilter(3)
|
+EagerAggregation(1)
|
+UNWIND
|
+ColumnFilter(4)
|
+Extract(3)
|
+ExtractPath
|
+Filter(1)
|
+TraversalMatcher
+---------------------+---------+----------+------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
| Operator | Rows | DbHits | Identifiers | Other |
+---------------------+---------+----------+------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
| ColumnFilter(0) | 10 | 0 | EXTRACT(v in NODES(p) | v.Page), count(p) | keep columns EXTRACT(v in NODES(p) | v.Page), count(p) |
| Extract(0) | 10 | 0 | FRESHID225, FRESHID258, EXTRACT(v in NODES(p) | v.Page), count(p) | EXTRACT(v in NODES(p) | v.Page), count(p) |
| ColumnFilter(1) | 10 | 0 | FRESHID225, FRESHID258 | keep columns , |
| Top | 10 | 0 | FRESHID225, INTERNAL_AGGREGATEf7fa022b-cdb5-4ef2-bec5-a2f4f10706b6 | { AUTOINT0}; Cached( INTERNAL_AGGREGATEf7fa022b-cdb5-4ef2-bec5-a2f4f10706b6 of type Integer) |
| EagerAggregation(0) | 212828 | 0 | FRESHID225, INTERNAL_AGGREGATEf7fa022b-cdb5-4ef2-bec5-a2f4f10706b6 | |
| Extract(1) | 1749120 | 10494720 | FRESHID225, distinct_views, p | |
| ColumnFilter(2) | 1749120 | 0 | distinct_views, p | keep columns distinct_views, p |
| Filter(0) | 1749120 | 0 | FRESHID196, distinct_views, p | CoercedPredicate(anon[196]) |
| Extract(2) | 2115766 | 0 | FRESHID196, distinct_views, p | |
| ColumnFilter(3) | 2115766 | 0 | distinct_views, p | keep columns p, distinct_views |
| EagerAggregation(1) | 2115766 | 0 | INTERNAL_AGGREGATEb0939c81-a40c-4012-afd6-4852b17cf2e4, p | p |
| UNWIND | 6347298 | 0 | p, pages, views | |
| ColumnFilter(4) | 2115766 | 0 | p, pages | keep columns p, pages |
| Extract(3) | 2115766 | 12694596 | p, pages | pages |
| ExtractPath | 2115766 | 0 | p | |
| Filter(1) | 2115766 | 2115766 | | hasLabel(anon[34]:PageView(0)) |
| TraversalMatcher | 2115766 | 16926150 | | , , , |
+---------------------+---------+----------+------------------------------------------------------------------------+------------------------------------------------------------------------------------------------+
Total database accesses: 42231232
2.
match (p1:PageView)-[:Next]->(p2:PageView)-[:Next]->(p3:PageView)
where p1.Page<>p2.Page and p1.Page<>p3.Page and p2.Page<>p3.Page
RETURN [p1.Page,p2.Page,p3.Page], count(*) as count
ORDER BY count DESC
LIMIT 10;
profile output:
10 rows
28660 ms
Compiler CYPHER 2.2-cost
Projection(0)
|
+Top
|
+EagerAggregation
|
+Projection(1)
|
+Filter(0)
|
+Expand(0)
|
+Filter(1)
|
+Expand(1)
|
+NodeByLabelScan
+------------------+---------------+---------+----------+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Operator | EstimatedRows | Rows | DbHits | Identifiers | Other |
+------------------+---------------+---------+----------+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection(0) | 1241 | 10 | 0 | FRESHID146, [p1.Page,p2.Page,p3.Page], count | [p1.Page,p2.Page,p3.Page], count |
| Top | 1241 | 10 | 0 | FRESHID146, count | { AUTOINT0}; count |
| EagerAggregation | 1241 | 212828 | 0 | FRESHID146, count | |
| Projection(1) | 1542393 | 1749120 | 10494720 | FRESHID146, p1, p2, p3 | |
| Filter(0) | 1542393 | 1749120 | 17872173 | p1, p2, p3 | (((hasLabel(p3:PageView(0)) AND NOT(Property(p1,Page(3)) == Property(p3,Page(3)))) AND NOT(anon[20] == anon[43])) AND NOT(Property(p2,Page(3)) == Property(p3,Page(3)))) |
| Expand(0) | 1904189 | 1985797 | 3971596 | p1, p2, p3 | (p2)-[:Next]->(p3) |
| Filter(1) | 1904191 | 1985799 | 10578840 | p1, p2 | (NOT(Property(p1,Page(3)) == Property(p2,Page(3))) AND hasLabel(p2:PageView(0))) |
| Expand(1) | 2115767 | 2115768 | 4231538 | p1, p2 | (p1)-[:Next]->(p2) |
| NodeByLabelScan | 2115770 | 2115770 | 2115771 | p1 | :PageView |
+------------------+---------------+---------+----------+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3. (With loops!? And I don't know why! I suggested that if identifiers are different then nodes are different)
match (pv1:PageView)-[:Next]->(pv2:PageView)-[:Next]->(pv3:PageView),
(pv1)-[:At]->(p1),(pv2)-[:At]->(p2),(pv3)-[:At]->(p3)
RETURN [p1.Name,p2.Name,p3.Name], count(*) as count
ORDER BY count DESC
LIMIT 10;
profile output:
10 rows
27678 ms
Compiler CYPHER 2.2-cost
Projection(0)
|
+Top
|
+EagerAggregation
|
+Projection(1)
|
+Filter(0)
|
+Expand(0)
|
+Filter(1)
|
+Expand(1)
|
+Filter(2)
|
+Expand(2)
|
+Filter(3)
|
+Expand(3)
|
+Expand(4)
|
+NodeByLabelScan
+------------------+---------------+---------+----------+------------------------------------------------+------------------------------------------------------------+
| Operator | EstimatedRows | Rows | DbHits | Identifiers | Other |
+------------------+---------------+---------+----------+------------------------------------------------+------------------------------------------------------------+
| Projection(0) | 1454 | 10 | 0 | FRESHID139, [p1.Name,p2.Name,p3.Name], count | [p1.Name,p2.Name,p3.Name], count |
| Top | 1454 | 10 | 0 | FRESHID139, count | { AUTOINT0}; count |
| EagerAggregation | 1454 | 223557 | 0 | FRESHID139, count | |
| Projection(1) | 2115760 | 2115764 | 12694584 | FRESHID139, p1, p2, p3, pv1, pv2, pv3 | |
| Filter(0) | 2115760 | 2115764 | 0 | p1, p2, p3, pv1, pv2, pv3 | (NOT(anon[116] == anon[80]) AND NOT(anon[80] == anon[98])) |
| Expand(0) | 2115760 | 2115764 | 4231530 | p1, p2, p3, pv1, pv2, pv3 | (pv1)-[:At]->(p1) |
| Filter(1) | 2115762 | 2115766 | 2115766 | p2, p3, pv1, pv2, pv3 | (hasLabel(pv1:PageView(0)) AND NOT(anon[21] == anon[45])) |
| Expand(1) | 2115762 | 2115766 | 4231532 | p2, p3, pv1, pv2, pv3 | (pv2)<-[:Next]-(pv1) |
| Filter(2) | 2115764 | 2115766 | 0 | p2, p3, pv2, pv3 | NOT(anon[116] == anon[98]) |
| Expand(2) | 2115764 | 2115766 | 4231534 | p2, p3, pv2, pv3 | (pv2)-[:At]->(p2) |
| Filter(3) | 2115766 | 2115768 | 2115768 | p3, pv2, pv3 | hasLabel(pv2:PageView(0)) |
| Expand(3) | 2115765 | 2115768 | 4231536 | p3, pv2, pv3 | (pv3)<-[:Next]-(pv2) |
| Expand(4) | 2115767 | 2115768 | 4231538 | p3, pv3 | (pv3)-[:At]->(p3) |
| NodeByLabelScan | 2115770 | 2115770 | 2115771 | pv3 | :PageView |
+------------------+---------------+---------+----------+------------------------------------------------+------------------------------------------------------------+
System info:
windows 8.1
250G ssd
neo4j enterprise 2.2.0-M02
cache: hpc
ram: 8G
jvm heap size: 4G
memory mapping: 50%
149 (:Page) nodes
2115770 (:PageView) nodes
Why even the fastest of this three methods is so slow? (I guess that all my data is in RAM)
What is the best way to filter paths with loops?
By specifying labels for all identifiers, you force Cypher to open the node headers and filter all labels in it.
This is where the names of your relationships are important. Relationships are made to drive you into the graph, for performance there would be no need to specify the labels, so if your sure all nodes along the Path have the Pageview label, just omit it except for the start of your query :
match (p1:PageView)-[:Next]->(p2)-[:Next]->(p3)
where p1.Page<>p2.Page and p1.Page<>p3.Page and p2.Page<>p3.Page
RETURN [p1.Page,p2.Page,p3.Page], count(*) as count
ORDER BY count DESC
LIMIT 10;
I posted some query plan results in this answer related to your question : Neo4j: label vs. indexed property?

Resources