HiveSQLException: cannot recognize input near 'SELECT' 'MAX' '(' in expression specification - hadoop

I'm trying to get the maximum value of a count. The code is as follows
SELECT coachID, COUNT(coachID)
FROM coaches_awards GROUP BY coachID
HAVING COUNT(coachID) =
(
SELECT MAX(t2.awards)
FROM (
SELECT coachID, count(coachID) as awards
FROM coaches_awards
GROUP BY coachID
) t2
);
Yet something keeps failing. The inner query works and gives the answer that I want and the outer query will work if the inner query is replaced by the number required. So I'm assuming I've made some syntax error.
Where am I going wrong?

If you are just looking for one row, why not do:
SELECT coachID, COUNT(coachID) as cnt
FROM coaches_awards
GROUP BY coachID
ORDER BY cnt DESC
LIMIT 1;
If you want ties, then use RANK() or DENSE_RANK():
SELECT ca.*
FROM (SELECT coachID, COUNT(*) as cnt,
RANK() OVER (ORDER BY COUNT(*) DESC) as seqnum
FROM coaches_awards
GROUP BY coachID
) ca
WHERE seqnum = 1;

Related

ORA-00979 Not a Group function error for query with User defined function in select statement

I have this query where a user defined function is added in the select and group by statement.
The inner select query without the WITH clause runs fine and doesn't give any error. But after adding WITH clause it gives the following error -
ORA-00979: not a GROUP BY expression
00979. 00000 - "not a GROUP BY expression"
*Cause:
*Action: Error at Line: 3 Column: 29
I need the WITH clause to return only a subset of the entire result set based on input ranges.
Query is as follows:
WITH INFO AS (
SELECT
GET_EVAULATED_VALUE(T.C_IMP, T.IMP) AS IMPORTANCE,
count(*) AS NO_OF_PC_AFFECTED
FROM TABLE_NAME T
WHERE T.ACNT_REL_ID = 16
GROUP BY
(GET_EVAULATED_VALUE(T.C_IMP, T.IMP))
ORDER BY IMPORTANCE desc
)
SELECT * FROM
(
SELECT ROWNUM AS RN,
(SELECT COUNT(*) FROM INFO) COUNTS,
IMPORTANCE
FROM INFO
)
WHERE RN > 0 AND RN <= 10;
I am not sure how to use CTE with group by on user defined function. But I realized that I can rewrite the query to remove sub-query and CTE and make it simpler as following (and it works):
select * from (
select a.*, ROWNUM rnum from
(SELECT
count(*) over() as COUNTS,
GET_EVAULATED_VALUE(T.C_IMP, T.IMP) AS IMPORTANCE,
count(*) AS NO_OF_PC_AFFECTED
FROM TABLE_NAME T
WHERE T.ACNT_RELATION_ID = 16
GROUP BY
(GET_EVAULATED_VALUE(T.C_IMP, T.IMP))
ORDER BY importance desc) a
where ROWNUM <= 10 )
where rnum >= 0;
Same issue here, I created a table "TABLE_CTE" instead of using a CTE and it worked.
CREATE TABLE TABLE_CTE
AS
SELECT
USER_DEFINED_FUNCTION(date_1),
COUNT(*)
FROM
TABLE_NAME
GROUP BY
USER_DEFINED_FUNCTION(date_1)
;
SELECT * FROM TABLE_CTE

Adding filters in subquery from CTE quadruples run time

I am working on an existing query for SSRS report that focuses on aggregated financial aid data split out into 10 aggregations. User wants to be able to select students included in that aggregated data based on new vs. returning and 'selected for verification.' For the new/returning status, I added a CTE to return the earliest admit date for a student. 2 of the 10 data fields are created by a subquery. I have been trying for 3 days to get the subquery to use the CTE fields for a filter, but they won't work. Either they're ignored or I get a 'not a group by expression' error. If I put the join to the CTE within the subquery, the query time jumps from 45 second to 400 seconds. This shouldn't be that complicated! What am I missing? I have added some of the code... 3 of the chunks work - paid_something doesn't.
with stuStatus as
(select
person_uid, min(year_admitted) admit_year
from academic_study
where aid_year between :AidYearStartParameter and :AidYearEndParameter
group by person_uid)
--- above code added to get student information not originally in qry
select
finaid_applicant_status.aid_year
, count(1) as fafsa_cnt --works
, sum( --works
case
when (
package_complete_date is not null
and admit.status is not null
)
then 1
else 0
end
) as admit_and_package
, (select count(*) --does't work
from (
select distinct award_by_aid_year.person_uid
from
award_by_aid_year
where
award_by_aid_year.aid_year = finaid_applicant_status.aid_year
and award_by_aid_year.total_paid_amount > 0 )dta
where
(
(:StudentStatusParameter = 'N' and stuStatus.admit_year = finaid_applicant_status.aid_year)
OR
(:StudentStatusParameter = 'R' and stuStatus.admit_year <> finaid_applicant_status.aid_year)
OR :StudentStatusParameter = '%'
)
)
as paid_something
, sum( --works
case
when exists (
select
1
from
award_by_person abp
where
abp.person_uid = fafsa.person_uid
and abp.aid_year = fafsa.aid_year
and abp.award_paid_amount > 0
) and fafsa.requirement is not null
then 1
else 0
end
) as paid_something_fafsa
from
finaid_applicant_status
join finaid_tracking_requirement fafsa
on finaid_applicant_status.person_uid = fafsa.person_uid
and finaid_applicant_status.aid_year = fafsa.aid_year
and fafsa.requirement = 'FAFSA'
left join finaid_tracking_requirement admit
on finaid_applicant_status.person_uid = admit.person_uid
and finaid_applicant_status.aid_year = admit.aid_year
and admit.requirement = 'ADMIT'
and admit.status in ('M', 'P')
left outer join stuStatus
on finaid_applicant_status.person_uid = stuStatus.person_uid
where
finaid_applicant_status.aid_year between :AidYearStartParameter and :AidYearEndParameter
and (
(:VerifiedParameter = '%') OR
(:VerifiedParameter <> '%' AND finaid_applicant_status.verification_required_ind = :VerifiedParameter)
)
and
(
(:StudentStatusParameter = 'N' and (stuStatus.admit_year IS NULL OR stuStatus.admit_year = finaid_applicant_status.aid_year ))
OR
(:StudentStatusParameter = 'R' and stuStatus.admit_year <> finaid_applicant_status.aid_year)
OR :StudentStatusParameter = '%'
)
group by
finaid_applicant_status.aid_year
order by
finaid_applicant_status.aid_year
Not sure if this helps, but you have something like this:
select aid_year, count(1) c1,
(select count(1)
from (select distinct person_uid
from award_by_aid_year a
where a.aid_year = fas.aid_year))
from finaid_applicant_status fas
group by aid_year;
This query throws ORA-00904 FAS.AID_YEAR invalid identifier. It is because fas.aid_year is nested too deep in subquery.
If you are able to modify your subquery from select count(1) from (select distinct sth from ... where year = fas.year) to select count(distinct sth) from ... where year = fas.year then it has the chance to work.
select aid_year, count(1) c1,
(select count(distinct person_uid)
from award_by_aid_year a
where a.aid_year = fas.aid_year) c2
from finaid_applicant_status fas
group by aid_year
Here is simplified demo showing non-working and working queries. Of course your query is much more complicated, but this is something what you could check.
Also maybe you can use dbfiddle or sqlfiddle to set up some test case? Or show us sample (anonimized) data and required output for them?

ORA-00904 invalid identifier for nested select

I have been stuck on this error for 2 hours.
I have nested select to get the first value.
select tbl.table_name,
(select distinct(FirstItem)
from
(select first_value(column_name) over (order by timestamp asc rows unbounded predecing) as FirstItem
from log_table_b l
where tbl.assignment_no = l.rpt_no)
) as "USERNAME",
from prod_table tbl;
It returns this error:
ERROR at line 6:
ORA-00904: "TBL"."ASSIGNMENT_NO": invalid identifier
I have tried many things, none of them seems to be helping me.
You can't use the parent table in inner sub query. Here how you could achieve this :
with tmp_table as
(
select rpt_no, first_value(column_name) over (order by timestamp asc rows unbounded predecing) as FirstItem
from log_table_b l
) select distinct tbl.table_name, firstItem
from prod_table tbl
join tmpTable on tmp_table.rpt_no = tbl.assignment_no;
You might want to find a more descriptive name to tmp_table
The issue is that you can only pass a reference from an outer query down to the next subquery level.
Here are a couple of alternatives:
select tbl.table_name,
(select min(column_name) keep (dense_rank first order by tstamp asc)
from log_table_b l
where tbl.assignment_no = l.rpt_no
) as "USERNAME"
from prod_table tbl;
select tbl.table_name,
l.username
from prod_table tbl
inner join (select rpt_no,
min(column_name) keep (dense_rank first order by tstamp asc) username
from log_table_b l
group by rpt_no)
on (tbl.assignment_no = l.rpt_no);
N.B. untested

RANK OVER function in Hive

I'm trying to run this query in Hive to return only the top 10 url which appear more often in the adimpression table.
select
ranked_mytable.url,
ranked_mytable.cnt
from
( select iq.url, iq.cnt, rank() over (partition by iq.url order by iq.cnt desc) rnk
from
( select url, count(*) cnt
from store.adimpression ai
inner join zuppa.adgroupcreativesubscription agcs
on agcs.id = ai.adgroupcreativesubscriptionid
inner join zuppa.adgroup ag
on ag.id = agcs.adgroupid
where ai.datehour >= '2014-05-15 00:00:00'
and ag.siteid = 1240
group by url
) iq
) ranked_mytable
where
ranked_mytable.rnk <= 10
order by
ranked_mytable.url,
ranked_mytable.rnk desc
;
Unfortunately I get an error message stating:
FAILED: SemanticException [Error 10002]: Line 26:23 Invalid column reference 'rnk'
I've tried to debug it and until the ranked_mytable sub-queries everything goes smooth. I've tried to comment the where ranked_mytable.rnk <= 10 clause but the error message keep appearing.
Hive is unable to order by a column that is not in the "output" of a select statement. To fix it, just include that column in the selected columns:
select
ranked_mytable.url,
ranked_mytable.cnt,
ranked_mytable.rnk
from
( select iq.url, iq.cnt, rank() over (partition by iq.url order by iq.cnt desc) rnk
from
( select url, count(*) cnt
from store.adimpression ai
inner join zuppa.adgroupcreativesubscription agcs
on agcs.id = ai.adgroupcreativesubscriptionid
inner join zuppa.adgroup ag
on ag.id = agcs.adgroupid
where ai.datehour >= '2014-05-15 00:00:00'
and ag.siteid = 1240
group by url
) iq
) ranked_mytable
where
ranked_mytable.rnk <= 10
order by
ranked_mytable.url,
ranked_mytable.rnk desc
;
If you don't want that 'rnk' column in your final output, I expect you could wrap that whole thing in another inner-query and just select out the 'url' and 'cnt' fields.
RANK OVER is not the best function to achieve this goal.
A better solution would be to use a combination of SORT BY and LIMIT. It's true in fact LIMIT picks randomly the rows in a table, but this might be avoided if used with the SORT BY function. From the Apache Wiki:
-- Top k queries. The following query returns the top 5 sales records wrt amount.
SET mapred.reduce.tasks = 1 SELECT * FROM sales SORT BY amount
DESC LIMIT 5
The query can be re-written in this way:
select
iq.url,
iq.cnt
from
( select url, count(*) cnt
from store.adimpression ai
inner join zuppa.adgroupcreativesubscription agcs
on agcs.id = ai.adgroupcreativesubscriptionid
inner join zuppa.adgroup ag
on ag.id = agcs.adgroupid
where ai.datehour >= '2014-05-15 00:00:00'
and ag.siteid = 1240
group by url ) iq
sort by
iq.cnt desc
limit
10
;
Remove the partition by iq.url clause from rank over() and re-run query.
Thanks & Regards,
Kamleshkumar Gujarathi
Put as before the rnk variable. It should work fine.

No Results returned for ROW_NUMBER() query

I am getting "no results returned" for the following query:
SELECT
Referer
FROM
(SELECT
ROW_NUMBER() OVER (ORDER BY CT.Referer ASC) AS RowNum,
CT.Referer, CT.LastModified
FROM
ClickTrack CT
JOIN
OrderTrack OT ON OT.ClickTrackID = CT.ClickTrackID
GROUP BY
CT.Referer, CT.LastModified
HAVING
LEN(CT.Referer) > 0) as num
WHERE
RowNum = 1
AND LastModified BETWEEN '07/06/2013' and '08/05/2013'
Curiously, when I leave off RowNum = 1, I get the full list of values. I need to get one at a time though to assign to a variable and drop into a temporary table.
The end query will be in a while loop using scalar variables in place of the date ranges and RowNum comparison.
Any help is appreciated. Thank you!
I'm thinking RowNum 1 may not have a date between your selections. Maybe put the date selection inside so that you know that the first one matches.
SELECT Referer
FROM (SELECT ROW_NUMBER() OVER (ORDER BY CT.Referer ASC)
AS RowNum, CT.Referer, CT.LastModified
FROM ClickTrack CT
JOIN OrderTrack OT ON OT.ClickTrackID = CT.ClickTrackID
WHERE CT.LastModified BETWEEN '07/06/2013' and '08/05/2013'
GROUP BY CT.Referer, CT.LastModified
HAVING LEN(CT.Referer) > 0) as num
WHERE RowNum = 1

Resources