This question already has answers here:
Does SparkSQL support subquery?
(2 answers)
Closed 6 years ago.
When I am running this query i got this type of error
select * from raw_2 where ip NOT IN (select * from raw_1);
org.apache.spark.sql.AnalysisException:
Unsupported language features in query:
select * from raw_2 where ip NOT IN (select * from raw_1)
TOK_QUERY 1, 0,24, 14
TOK_FROM 1, 4,6, 14
TOK_TABREF 1, 6,6, 14
TOK_TABNAME 1, 6,6, 14
raw_2 1, 6,6, 14
TOK_INSERT 0, -1,24, 0
TOK_DESTINATION 0, -1,-1, 0
TOK_DIR 0, -1,-1, 0
TOK_TMP_FILE 0, -1,-1, 0
TOK_SELECT 0, 0,2, 0
TOK_SELEXPR 0, 2,2, 0
TOK_ALLCOLREF 0, 2,2, 0
TOK_WHERE 1, 8,24, 29
NOT 1, 10,24, 29
TOK_SUBQUERY_EXPR 1, 14,10, 33
TOK_SUBQUERY_OP 1, 14,14, 33
IN 1, 14,14, 33
TOK_QUERY 1, 16,24, 51
TOK_FROM 1, 21,23, 51
TOK_TABREF 1, 23,23, 51
TOK_TABNAME 1, 23,23, 51
raw_1 1, 23,23, 51
TOK_INSERT 0, -1,19, 0
TOK_DESTINATION 0, -1,-1, 0
TOK_DIR 0, -1,-1, 0
TOK_TMP_FILE 0, -1,-1, 0
TOK_SELECT 0, 17,19, 0
TOK_SELEXPR 0, 19,19, 0
TOK_ALLCOLREF 0, 19,19, 0
TOK_TABLE_OR_COL 1, 10,10, 26
ip 1, 10,10, 26
scala.NotImplementedError: No parse rules for ASTNode type: 817, text:
TOK_SUBQUERY_EXPR :
TOK_SUBQUERY_EXPR 1, 14,10, 33
TOK_SUBQUERY_OP 1, 14,14, 33
IN 1, 14,14, 33
TOK_QUERY 1, 16,24, 51
TOK_FROM 1, 21,23, 51
TOK_
Spark 2.0.0+:
since 2.0.0 Spark supports a full range of subqueries. See Does SparkSQL support subquery? for details.
Spark < 2.0.0
Does Spark support subqqueries?
Generally speaking it does. Constructs like SELECT * FROM (SELECT * FROM foo WHERE bar = 1) as tmp perfectly valid queries in the Spark SQL.
As far as I can tell from the Catalyst parser source it doesn't support inner queries in a NOT IN clause:
| termExpression ~ (NOT ~ IN ~ "(" ~> rep1sep(termExpression, ",")) <~ ")" ^^ {
case e1 ~ e2 => Not(In(e1, e2))
}
It is still possible to use outer join followed by filter to obtain the same effect.
Related
I'm trying to implement a wavefront algorithm and I have a problem with the function, that produces the map with specific gradients.
I've tried several different versions of the code below and none of them worked.
The starting point for the algorithm (the goal) is set to 1 before and from that point on each neighbour's gradient should be increased (similar to every wavefront algorithm), if the gradient hasn't bin altered yet.
originX and originY is the goal, from which the alorithm should start. mapMatrix is a global variable
mapMatrix looks like this:
0 0 0 0 0 0 0
0 0 N 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 N 0 0 N 0 N
N N 0 0 N 0 0
0 0 0 0 0 0 0
(0 for rails, N(nil) for obstacles)
expected output example:
7 6 5 4 3 4 5
6 5 N 3 2 3 4
5 4 3 2 1 2 3
6 5 4 3 2 3 3
7 N 5 4 N 4 N
N N 6 5 N 5 6
9 8 7 6 7 6 7
And with this code for example:
function pathFinder(originX, originY)
northDir = originY - 1
eastDir = originX + 1
southDir = originY + 1
westDir = originX - 1
if northDir > 0 and mapMatrix[originX][northDir] == 0 then
mapMatrix[originX][northDir] = mapMatrix[originX][originY] + 1
pathFinder(originX, northDir)
end
if eastDir <= 7 and mapMatrix[eastDir][originY] == 0 then
mapMatrix[eastDir][originY] = mapMatrix[originX][originY] + 1
pathFinder(eastDir, originY)
end
if southDir <= 7 and mapMatrix[originX][southDir] == 0 then
mapMatrix[originX][southDir] = mapMatrix[originX][originY] + 1
pathFinder(originX, southDir)
end
if westDir > 0 and mapMatrix[westDir][originY] == 0 then
mapMatrix[westDir][originY] = mapMatrix[originX][originY] + 1
pathFinder(westDir, originY)
end
end
I get this mapMatrix:
0 0 0 0 3 4 5
0 0 N 0 2 10 6
0 0 0 0 1 9 7
0 0 0 0 0 0 8
0 N 0 0 N 0 N
N N 0 0 N 0 0
0 0 0 0 0 0 0
And if I switch the if statements arround it produces different version of mapMatrix
After making northDir, etc local the output looks like this: EDIT
33 24 23 22 3 4 5
32 25 N 21 2 11 6
31 26 27 20 1 10 7
30 29 28 19 20 9 8
31 N 29 18 N 10 N
N N 30 17 N 11 12
33 32 31 16 15 14 13
If more code or information is needed, I'd be happy to help
Your code is just wrong at all. As pathFinder is called recursively in the first check, it will be just going in that direction until any obstacle appears, and than going in the next direction, and so on.
BFS is actually a pretty simple algorithm. It can be easily implemented iteratively on a queue without any recursion as follow:
Put initial node to a queue;
Pop first node from the queue and process it;
Push unprocessed adjacent nodes to the end of the queue;
If queue is not empty, go to the step 2.
In Lua on a rectangular matrix it can be implemented in about two or three dozen of lines:
function gradient(matrix, originX, originY)
-- Create queue and put origin position and initial value to it.
local queue = { { originX, originY, 1 } }
repeat
-- Pop first position and value from the queue.
local x, y, value = unpack(table.remove(queue, 1))
-- Mark this position in the matrix.
matrix[y][x] = value
-- Check position to the top.
if y > 1 and matrix[y - 1][x] == 0 then
-- If it is not already processed, push it to the queue.
table.insert(queue, { x, y - 1, value + 1 })
end
-- Check position on the left.
if x > 1 and matrix[y][x - 1] == 0 then
table.insert(queue, { x - 1, y, value + 1 })
end
-- Check position to the bottom.
if y < #matrix and matrix[y + 1][x] == 0 then
table.insert(queue, { x, y + 1, value + 1 })
end
-- Check position on the right.
if x < #matrix[y] and matrix[y][x + 1] == 0 then
table.insert(queue, { x + 1, y, value + 1 })
end
-- Repeat, until queue is not empty.
until #queue == 0
end
-- Just helper function to print a matrix.
function printMatrix(matrix)
for _, row in pairs(matrix) do
for _, value in pairs(row) do
io.write(string.format("%2s", value))
end
io.write('\n')
end
end
local mapMatrix = {
{ 0, 0, 0, 0, 0, 0, 0, },
{ 0, 0, 'N', 0, 0, 0, 0, },
{ 0, 0, 0, 0, 0, 0, 0, },
{ 0, 0, 0, 0, 0, 0, 0, },
{ 0, 'N', 0, 0, 'N', 0, 'N', },
{ 'N', 'N', 0, 0, 'N', 0, 0, },
{ 0, 0, 0, 0, 0, 0, 0, },
}
gradient(mapMatrix, 5, 3)
printMatrix(mapMatrix)
--[[
Produces:
7 6 5 4 3 4 5
6 5 N 3 2 3 4
5 4 3 2 1 2 3
6 5 4 3 2 3 4
7 N 5 4 N 4 N
N N 6 5 N 5 6
9 8 7 6 7 6 7
]]
This is a complete script, runnable in the console.
But although, for illustrative purposes, this code is very simple, it is not very efficient. Each removal of the first item from the queue causes reindexing of the remaining items. For production code you should implement a linked list or something similar for the queue.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
. . 1 1 1 1 1 0 0 0 1 1 1 1 1 0 0 0
Recognition starts at 17 and goes backwards to 0.
What can be seen is the most simple pattern.
Pattern starts with at least three 0s or three 1s but could be more of each but not mixed!
First pattern is then followed by at least five 0s or five 1s depending on what came in the first pattern. Since pattern one contains three 0, there must be at least five 1s and vice versa
Then we want to see the first pattern again. At least three 0s or three 1s, again, depending wheather there were 1s or 0s before
Finally we want to see the second pattern again, which means at least five 0s or five 1s, again, depending on which pattern was seen before
I tried using for loops and counters but did not manage to work it out. What is struggling me is the fact, that the pattern is not of fixed size as there can be more than three or five 0s and 1s in succession.
Is anybody able to provide some pseudo code how to implement this or even some MQL5 code?
The following Swift code is everything else than optimal. It should just give you hints how you could implement it.
A function to match a single pattern:
func matchPattern(numbers: [Int], startIndex: Int, number: Int) -> Int {
var actualIndex = startIndex
while numbers[actualIndex] == number && actualIndex > 0 {
actualIndex = actualIndex - 1
}
return startIndex - actualIndex
}
A function to match the 4 patterns:
func match(binNrs: [Int]) -> Bool {
let firstPatternNr = binNrs[17]
let secondPatternNr = firstPatternNr == 0 ? 1 : 0
let pattern1Length = matchPattern(numbers: binNrs,
startIndex: 17,
number: firstPatternNr)
if pattern1Length < 3 { return false }
let pattern2Length = matchPattern(numbers: binNrs,
startIndex: 17 - pattern1Length,
number: secondPatternNr)
if pattern2Length < 5 { return false }
let pattern3Length = matchPattern(numbers: binNrs,
startIndex: 17 - pattern1Length - pattern2Length,
number: firstPatternNr)
if pattern3Length < 3 { return false }
let pattern4Length = matchPattern(numbers: binNrs,
startIndex: 17 - pattern1Length - pattern2Length - pattern3Length,
number: secondPatternNr)
return pattern4Length >= 5
}
Some test patterns with results:
let match1 = match(binNrs: [0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0]) // true
let match2 = match(binNrs: [1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]) // false (4th sequence < 5)
let match3 = match(binNrs: [0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0]) // false (1st sequence < 3)
let match4 = match(binNrs: [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1]) // false (2nd sequence < 5)
C1 C2 C3 C4 C5 C6 C7 C8 Total **Percentages**
======================================================
R1 6 1 8 8 2 1 1 0 27 **60%**
R2 0 0 0 5 1 1 0 0 7 **16%**
R3 2 0 3 2 0 1 0 0 8 **18%**
R4 2 0 0 1 0 0 0 0 3 **7%**
TTL10 1 11 16 3 3 1 0 45 **100%**
How to calculate the individual row percentages in SSRS
Thank you.
If you're not filtering your dataset, you could use the Dataset sum to get the overall total and use that as the denominator in your expression.
If your table is a matrix with the C1 - C8 all coming from one field, then your formula would just be:
=Sum(Fields!YourField.Value) / Sum(Fields!YourField.Value, "Dataset1")
If the C1 - C8 fields are in separate fields, you can use the same expression used for your total column as the numerator and then divide by the SUM of all the other fields.
=Sum(Fields!C1.Value + Fields!C2.Value + Fields!C3.Value + Fields!C4.Value + Fields!C5.Value + Fields!C6.Value + Fields!C7.Value + Fields!C8.Value)
/
Sum(Fields!C1.Value + Fields!C2.Value + Fields!C3.Value + Fields!C4.Value + Fields!C5.Value + Fields!C6.Value + Fields!C7.Value + Fields!C8.Value, "Dataset1"))
I will work on SQL rather on SSRS. Here is my approach. For SSRS here is the link.
DECLARE #YourTable TABLE
(
Col INT
,Col1 INT
,Col2 INT
,Col3 INT
)
INSERT INTO #YourTable VALUES
(1 , 20, 10, 15)
,(2 , 30, 12, 14)
,(2 , 22, 2, 4)
,(3 , 3, 10, 15)
,(5 , 5, 14, 14)
,(2 , 21, 32, 4)
SELECT * FROM #YourTable
; WITH CTE AS
(SELECT *,Col+Col1+Col2+Col3 AS SumCol FROM #YourTable)
SELECT *, CAST(SumCol*100.0 / SUM(SumCol) OVER() as DECIMAL(28,2)) FROM CTE
Here's another approach:
Create a row outside of the details group, above the first row of data.
Populate a Textbox in the new row =Sum(Fields!Total.Value). Rename the Textbox something unique, such as Denominator.
Hide the row.
For your percentage formula in the details row, use something like:
=Sum(Fields!Total.Value) / ReportItems!Denominator.Value
I am getting NaN in 3 places in my SSRS report. I believe it is because I am dividing by 0. I am trying to find the average days for prescriptions on-time, late, and not filled. The 3 expressions that I was given are below. What and where would I need to insert the iif statement addressing the 0 issue. I am new to this
=sum(iif(Fields!DaysDifference.Value >= -1 and Fields!DaysDifference.Value <= 1 and Fields!ActualNextFillDateKey.Value <> 0, Fields!DaysDifference.Value,0))/
sum(iif(Fields!DaysDifference.Value >= -1 and Fields!DaysDifference.Value <= 1 and Fields!ActualNextFillDateKey.Value <> 0, 1,0))
=sum(iif(Fields!DaysDifference.Value > 1 and Fields!ActualNextFillDateKey.Value <> 0, Fields!DaysDifference.Value,0))/
sum(iif(Fields!DaysDifference.Value > 1 and Fields!ActualNextFillDateKey.Value <> 0, 1,0))
=sum(iif(Fields!ActualNextFillDateKey.Value = 0, Fields!DaysDifference.Value, 0))/
sum(iif(Fields!ActualNextFillDateKey.Value = 0, 1, 0))
Instead of using 0, you should be dividing by your field:
=SUM(IIF(Fields!ActualNextFillDateKey.Value = 0, Fields!DaysDifference.Value, 0))/
SUM(IIF(Fields!ActualNextFillDateKey.Value = 0, 1, Fields!ActualNextFillDateKey.Value))
I need to be able to process one word or words and verify that it has valid syllables. There are some syllabification rules that could be used:
V CV VC CVC CCV CCCV CVCC
where V is a vowel and C is a consonant. e.g.,
pronunciation (5 Pro-nun-ci-a-tion; CCV-CVC-CV-V-CVC)
Or is there a simple code that can be used, or a library in c++? In class we're talking about binary search trees, hash tables, etc, but i can't really see the relation. Any help would appreciated, thanks.
Whenever we have collected a full pattern-string, we can either discard it and begin collecting to a new pattern-string, or keep it and try to get a longer pattern-string. We don't know in advance (without examining the rest of the input-string), whether we should keep or discard the current string, so we need to keep both possibilities in mind.
We can build a state machine that can keep track of this for us. The base-states are identified by the sequence of characters we have examined so far:
State C V
"" {"C"} {"V",""}
"C" {"CC"} {"CV",""}
"CC" {"CCC"} {""}
"CCC" {} {""}
"CV" {"CVC",""} {}
"CVC" {""} {}
"V" {""} {}
Since we don't always know which action to take, we can be in several possible states at once. Those sets of possible states form super-states:
Index Super-state C V
0 {} 0 0 Fail
1 {""} 2 9 Accept
2 {"C"} 3 8
3 {"CC"} 4 1
4 {"CCC"} 0 1
5 {"","C"} 6 13 Accept
6 {"C","CC"} 7 8
7 {"CC","CCC"} 4 1
8 {"","CV"} 12 9 Accept
9 {"","V"} 5 9 Accept
10 {"","C","CC"} 11 13 Accept
11 {"C","CC","CCC"} 7 8
12 {"","C","CVC"} 10 13 Accept
13 {"","CV","V"} 12 9 Accept
The transitions are between super-states. Each member of the super-state is advanced with the same symbol. All members without such transition are discarded. If a member has two possible destinations, both are added to the new super-state.
You might notice that some rows are very similar. Super-state 3 and 7 have the same transitions. As are 6 and 11, and 8 and 13. You could collapse those into one state each, and update the indices. I'm not going to demonstrate that here.
This could easily be encoded into a programming language:
// index = 0 1 2 3 4 5 6 7 8 9 10 11 12 13
int[] consonant = new int[] { 0, 2, 3, 4, 0, 6, 7, 4, 12, 5, 11, 7, 10, 12 };
int[] vocal = new int[] { 0, 9, 8, 1, 1, 13, 8, 1, 9, 9, 13, 8, 13, 9 };
int[] accept = new int[] { 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1 };
int startState = 1;
int failState = 0;
bool CheckWord(string word)
{
int state = startState;
foreach (char c in word)
{
if (IsVocal(c))
{
state = vocal[state];
}
else if (IsConsonant(c))
{
state = consonant[state];
}
if (state == failState) return false;
}
return accept[state] != 0;
}
Example:
> CheckWord("pronunciation")
true
> CheckWord("pronunciationn")
false