This might be answered somewhere else already, but I was looking for an answer the past few days and couldn't find an answer that suited my problem/that I understood...
I'm using CakePHP 3.8.5 and am currently working on a query that includes a subquery in the select.
I got 3 tables Locations, Computers and Printers. Locations and Computers are in a belongsToMany relationship, as well as Locations and Printers.
So I'm trying to get the following query, which is working well as far as the data results go:
$computersQuery = $this->Computers->find();
$computersQuery->select([$computersQuery->func()->count('*')])
->where(function (QueryExpression $exp) {
return $exp
->notEq('Computers.Unwanted_Software', '')
->equalFields('Computers.Area_ID', 'Locations.Area_ID');
});
$printersQuery = $this->Printers->find();
$printersQuery->select($printersQuery->func()->count('*'))
->where(function (QueryExpression $exp) {
return $exp
->eq('Printers.WHQL', 0)
->equalFields('Printers.Area_ID', 'Locations.Area_ID');
});
$dataQuery = $this->Locations->find();
$dataQuery->select(['Locations.Area_ID',
'Unwanted_Software' => $computersQuery,
'Printers_Not_WHQL_Compatible' => $printersQuery])
->group('Locations.Area_ID');
So I'm trying to paginate the $dataQuery in my Controller. In my model I can click all three column headers but only the Area_ID column will get sorted. The two subquery columns won't sort. Even tho I'm not getting Errors.
Looking at the SQL-log shows that it's never even trying to order by those two columns...
Any ideas how to fix this/work around this are highly appreciated!
If you need more info about my code just leave a comment below.
EDIT 1:
As user #ndm pointed out, I had to put the computed fields into the sortWhitelist option of the pagination array.
Doing that worked out well as I was able to sort by the column headers:
$this->paginate = [
'limit' => '100',
'sortWhitelist' => [
'Locations.Area_ID',
'Unwanted_Software',
'Printers_Not_WHQL_Compatible'
],
'order' => ['Locations.Area_ID' => 'ASC']
]
But then the next problem appeared. Was trying to sort by the Printers_Not_WHQL_Compatible column. The generated SQL code had one small issue (BTW I'm using SQL Server 2008):
SELECT *
FROM (SELECT Locations.Area_ID AS [Locations__Area_ID],
(SELECT (COUNT(*)) FROM computers Computers WHERE (...)) AS [Unwanted_Software],
(SELECT (COUNT(*)) FROM printers Printers WHERE (...)) AS [Printers_Not_WHQL_Compatible],
(ROW_NUMBER() OVER (ORDER BY SELECT (COUNT(*)) FROM printers Printers WHERE (...) asc, Locations.Area_ID ASC)) AS [_cake_page_rownum_] FROM locations_Views Locations GROUP BY Locations.Area_ID ) _cake_paging_
WHERE _cake_paging_._cake_page_rownum_ <= 100
This represents the generated SQL code. Problem is that in de Order By statement there are no brackets around my subquery. It should look like this, so that SQL Server can read it:
... ORDER BY ( SELECT (COUNT(*)) FROM printers Printers WHERE (...) ) asc, ...
Any ideas how to fix this?
EDIT 2:
So #ndm answer with either the pull-request or the fix via the newExpr() function both work great. At least regarding the parentheses for the subquery in the Order By.
Sadly already running into the next problem. The generated SQL (this counts for both solutions!) is kinda "refreshing" the parameter input for the entire query in the Order By, which means that it puts the filter parameters for the Where clauses in starting again by parameter :c0. You can see that in the following query:
SELECT *
FROM (SELECT Locations.Area_ID AS [Locations__Area_ID],
(SELECT (COUNT(*))
FROM computers Computers
WHERE (Computers.Unwanted_Software != :c0
AND Computers.Area_ID = (Locations.Area_ID)
)
) AS [Unwanted_Software],
(SELECT (COUNT(*))
FROM printers Printers
WHERE (Printers.WHQL = :c1
AND Printers.Area_ID = (Locations.Area_ID))
) AS [Printers_Not_WHQL_Compatible],
(ROW_NUMBER() OVER (ORDER BY
(SELECT (COUNT(*))
FROM printers Printers
WHERE (Printers.WHQL = :c0
AND Printers.Area_ID = (Locations.Area_ID))) asc,
Locations.Area_ID ASC)
) AS [_cake_page_rownum_]
FROM locations_Views Locations
GROUP BY Locations.Area_ID ) _cake_paging_
WHERE _cake_paging_._cake_page_rownum_ <= 100
I don't think this is the intended result. I personally can probably work around that, by avoiding passing parameters directly (I don't have to deal with concrete external user input). Still think this should be looked over.
Thanks #ndm for great help! Definitely an upvote from me.
As linked in the comments, by default the paginator only allows sorting on columns that exist in the main table, columns of other tables (joins) or computed columns must be explicitly allowed via the sortWhiteList option.
The missing parentheses in the the generated window function SQL looks like a bug, the SQL Server query translator doesn't check whether the expression that is defined in the order clause is a query, which would require to wrap the generated SQL in parentheses.
I've pushed a possible fix for 3.9 and 4.1:
https://github.com/cakephp/cakephp/pull/15165
https://github.com/cakephp/cakephp/pull/15164
If you cannot upgrade right now, a possible workaround could be to wrap your subqueries in additional expressions, which when compiled should wrap the inner query expression in parentheses:
$dataQuery
->select([
'Locations.Area_ID',
'Unwanted_Software' => $dataQuery->newExpr($computersQuery),
'Printers_Not_WHQL_Compatible' => $dataQuery->newExpr($printersQuery)
])
->group('Locations.Area_ID');
Related
I have a fairly large and complex query being generated by Yii. To generate this query we're utilizing CDbCritera::with to eagerly load multiple related models, and we're using multiple scopes to limit the records returned. The query being generated is roughly 700 lines long, but looks something like this:
SELECT `t`.`column1` as `t0_c0`,
`t`.`column2` as `t0_c1`,
`related1`.`column1` as `t1_c0`,
...
`related9`.`column5` as `t9_c4`
FROM `model` `t`
LEFT OUTER JOIN `other_model` `related1`
ON ( `t`.`other_model_id` = `related1`.`id` )
...
LEFT OUTER JOIN `more_models` `related9`
ON ( `t`.`more_models_id` = `related9`.`id` )
WHERE
...big long WHERE clause using all of related1 - related9 to filter model...
LIMIT 10
Our database has a not insignificant amount of data, but not obscene, either. In this case the model table has about 126000 rows, every "related" model is a BELONGS_TO relationship and there is an index on t.XXX_id so the join is fairly trivial. The problem is the complexity of the WHERE clause, possessing multiple COALESCE and IF and CASE clauses. Performing the filter on our 126000 rows is taking 2.6 seconds -- far longer than we would like for an API endpoint.
The WHERE clause is divided into multiple different sections like so:
WHERE
( ... part 1 ... )
AND
( ... part 2 ... )
AND
( ... part 3 ... )
With each part corresponding to one of the scopes, and each part using one or more related models
One of the scopes filters on only a single related model, and in doing so filters our table down from 126000 rows to about 2000 rows. I found experimentally (in MySQL Workbench) that I could get our query from 2.6 seconds to 0.2 seconds by simply doing this:
SELECT `t`.`column1` as `t0_c0`,
`t`.`column2` as `t0_c1`,
`related1`.`column1` as `t1_c0`,
...
`related9`.`column5` as `t9_c4`
FROM
(
SELECT `model`.*
FROM `model`
LEFT OUTER JOIN `other_model`
ON ( `t`.`other_model_id` = `other_model`.`id` )
WHERE
( ... part 1 ... )
) `t`
LEFT OUTER JOIN `other_model` `related1`
ON ( `t`.`other_model_id` = `related1`.`id` )
...
LEFT OUTER JOIN `more_models` `related9`
ON ( `t`.`more_models_id` = `related9`.`id` )
WHERE
( ... part 2 ... )
AND
( ... part 3 ... )
LIMIT 10
This way instead of performing the very complex WHERE clause on all 126000 rows of the original model table, we perform the much simpler (and index-enhanced) WHERE clause on these 126000 rows and then perform the complex WHERE clause on only the 2000 relevant rows. The results of the two queries are identical, but using a subquery in the FROM clause causes it to run 13x faster.
The problem is, I have no idea how to do this in Yii. I know that I can use CDbCommand to build a query and even pass in raw SQL, but what I'll get back is an array of "rows" -- they won't be understood by Yii and properly converted to the right models.
Does Yii's ActiveRecord system have a way to say something like the following?
$criteria = new CDbCriteria;
$criteria->scopes = array("part1");
$subQuery = Model::model()->buildQuery($criteria);
$criteria = new CDbCriteria;
$criteria->scopes = array("part2", "part3");
$fullQuery = $subQuery->findAll($criteria);
Although not a perfect solution, I did find something that's almost as good. Break the original query into two:
Get the IDs or models you wish to select in the FROM subquery
Append a WHERE to the outer query with id in (...)
If anyone is interested I'll hunt down the code I wrote for this to post in the answer as an example, but so far this question has gotten very little attention and once I found a pseudo-decent solution I've moved on.
I'm working with my DBA to try to figure out a way to roll up all costs associated with a Work Order. Since any work Order can have multiple child work orders (through multiple "generations") as well as related work orders (through the RELATEDRECORDS table), I need to be able to get the total of the ACTLABORCOST and ACTMATERIALCOST fields for all child and related work orders (as well as each of their child and related work orders). I've worked though a hierarchical query (using CONNECT BY PRIOR) to get all the children, grandchildren, etc., but I'm stuck on the related work orders. Since every work order can have a related work order with it's own children and related work orders, I need an Oracle function that drills down through the children and the related work orders and their children and related work orders. Since I would think that this is something that should be fairly common, I'm hoping that there is someone who has done this and can share what they've done.
Another option would be a recursive query, as suggested by Francisco Sitja. Since my Oracle didn't allow 2 UNION ALLs, I had to joint to the WOANCESTOR table in both child queries instead of dedicating a UNION ALL for doing the WO hierarchy. I was then able to use the one permitted UNION ALL for doing the RELATEDRECORD hierarchy. And it seems to run pretty quickly.
with mywos (wonum, parent, taskid, worktype, description, origrecordid, woclass, siteid) as (
-- normal WO hierarchy
select wo.wonum, wo.parent, wo.taskid, wo.worktype, wo.description, wo.origrecordid, wo.woclass, wo.siteid
from woancestor a
join workorder wo
on a.wonum = wo.wonum
and a.siteid = wo.siteid
where a.ancestor = 'MY-STARTING-WONUM'
union all
-- WO hierarchy associated via RELATEDRECORD
select wo.wonum, wo.parent, wo.taskid, wo.worktype, wo.description, wo.origrecordid, wo.woclass, wo.siteid
from mywos
join relatedrecord rr
on mywos.woclass = rr.class
and mywos.siteid = rr.siteid
and mywos.wonum = rr.recordkey
-- prevent cycle / going back up the hierarchy
and rr.relatetype not in ('ORIGINATOR')
join woancestor a
on rr.relatedrecsiteid = a.siteid
and rr.relatedreckey = a.ancestor
join workorder wo
on a.siteid = wo.siteid
and a.wonum = wo.wonum
)
select * from mywos
;
Have you considered the WOGRANDTOTAL object? Its description in MAXOBJECT is "Non-Persistent table to display WO grandtotals". There is a dialog in the Work Order Tracking application that you can get to from the Select Action / More Actions menu. Since you mentioned it repeatedly, I should note that WOGRANDTOTAL values do not include joins across RELATEDRECORDS to other work order hierarchies.
You can also save yourself the complication of CONNECT BY PRIOR by joining to WOANCESTOR, which is effectively a dump from a CONNECT BY PRIOR query. (There are other %ANCESTOR tables for other hierarchies.)
I think a recursive automation script would be the best way to do what you want, if you need the results in Maximo. If you need the total cost outside of Maximo, maybe a recursive function would work.
We finally figured out how to pull this off.
WITH WO(WONUM,
PARENT) AS
((SELECT X.WONUM,
X.PARENT
FROM (SELECT R.RECORDKEY WONUM,
R.RELATEDRECKEY PARENT
FROM MAXIMO.RELATEDRECORD R
WHERE R.RELATEDRECKEY = '382418'
UNION ALL
SELECT W.WONUM,
W.PARENT
FROM MAXIMO.WORKORDER W
START WITH W.PARENT = '382418'
CONNECT BY PRIOR W.WONUM = W.PARENT) X)
UNION ALL
SELECT W.WONUM, W.PARENT FROM MAXIMO.WORKORDER W, WO WHERE W.WONUM = WO.PARENT)
SELECT DISTINCT WONUM FROM WO;
This returns a list of all of the child and related work orders for a given work order.
I'm trying to build a query that shows only non-unique duplicates. I've already built a query that shows all the records coming into consideration:
SELECT tbl_tm.title, lp_index.starttime, musicsound.archnr
FROM tbl_tm
INNER JOIN musicsound on tbl_tm.fk_tbl_tm_musicsound = musicsound.pk_musicsound
INNER JOIN lp_index ON musicsound.pk_musicsound = lp_index.fk_index_musicsound
INNER JOIN plan ON lp_index.fk_index_plan = plan.pk_plan
WHERE tbl_tm.FK_tbl_tm_title_type_music = '22' AND plan.airdate
BETWEEN to_date ('15-01-13') AND to_date('17-01-13')
GROUP BY tbl_tm.title, lp_index.starttime, musicsound.archnr
HAVING COUNT (tbl_tm.title) > 0;
The corresponding result set looks like this:
title starttime archnrr
============================================
Pumped up kicks 05:05:37 0616866
People Help The People 05:09:13 0620176
I can't dance 05:12:43 0600109
Locked Out Of Heaven 05:36:08 0620101
China in your hand 05:41:33 0600053
Locked Out Of Heaven 08:52:50 0620101
It gives me music titles played between a certain timespan along with their starting time and archive ID.
What I want to achieve is something like this:
title starttime archnr
============================================
Locked Out Of Heaven 05:36:08 0620101
Locked Out Of Heaven 08:52:50 0620101
There would only be two columns left: both share the same title and archive number but differ in the time part. Increasing the 'HAVING COUNT' value will give me a zero-row
result set, since there aren't any entries that are exactly the same.
What I've found out so far is that the solution for this problem will most likely have a nested subquery, but I can't seem to get it done. Any help on this would be greatly appreciated.
Note: I'm on a Oracle 11g-server. My user has read-only privileges. I use SQL Developer on my workstation.
You can try something like this:
SELECT title, starttime, archnr
FROM (
SELECT title, starttime, archnr, count(*) over (partition by title) cnt
FROM (your_query))
WHERE cnt > 1
Here is a sqlfiddle demo
Hi Can any one help me out of this query forming logic
SELECT C.CPPID, c.CPP_AMT_MANUAL
FROM CPP_PRCNT CC,CPP_VIEW c
WHERE
CC.CPPYR IN (
SELECT C.YEAR FROM CPP_VIEW_VIEW C WHERE UPPER(C.CPPNO) = UPPER('123')
AND C.CPP_CODE ='CPP000000000053'
and TO_CHAR(c.CPP_DATE,'YYYY/Mon')='2012/Nov'
)
AND UPPER(C.CPPNO) = UPPER('123')
AND C.CPP_CODE ='CPP000000000053'
and TO_CHAR(c.CPP_DATE,'YYYY/Mon') = '2012/Nov';
Please Correct me if i formed wrong query structure, in terms of query Performance and Standards. Thanks in Advance
If you have some indexes or partitioned tables I would not use functions on columns but on variables, to be able to use indexes/select partitions.
Also I use ANSI 92 SQL syntax. You don't specify(or not directly) a join contition between cpp_prcnt and cpp_view so it is actually a cartesian product(cross join)
SELECT C.CPPID, c.CPP_AMT_MANUAL
FROM CPP_PRCNT CC
CROSS JOIN CPP_VIEW c
WHERE
CC.CPPYR IN (
SELECT C.YEAR
FROM CPP_VIEW_VIEW C
WHERE C.CPPNO = '123'
AND C.CPP_CODE ='CPP000000000053'
AND trunc(c.CPP_DATE,'MM')=to_date('2012/Nov','YYYY/Mon')
)
AND C.CPPNO = '123'
AND C.CPP_CODE ='CPP000000000053'
AND trunc(c.CPP_DATE,'MM')=to_date('2012/Nov','YYYY/Mon')
If you show us the definition of cpp_view_view(seems to be a view over cpp_view), the definition(if simple) of CPP_VIEW and what you're trying to achieve, I bet there are more things to be improved/fixed.
There are a couple of things you could improve:
if possible, get rid of the UPPER() in the comparison - this will render any indices useless. If that's not possible, consider a function-based index on UPPER(CPPNO)
do not convert your DATE column to a string to compare it with a string - do it the other way round (i.e. convert your string to a date => only one conversion needed instead of one per table row, use of indices possible)
play around with EXISTS instead of IN, as suggested by Dileep - might be faster
I have a database with customers orders.
I want to use Linq (to EF) to query the db to bring back the last(most recent) 3,4...n orders for every customer.
Note:
Customer 1 may have just made 12 orders in the last hr; but customer 2 may not have made any since last week.
I cant for the life of me work out how to write query in linq (lambda expressions) to get the data set back.
Any good ideas?
Edit:
Customers and orders is a simplification. The table I am querying is actually a record of outbound messages to various web services. It just seemed easer to describe as customers and orders. The relationship is the same.
I am building a task that checks the last n messages for each web service to see if there were any failures. We are wanting a semi real time Health status of the webservices.
#CoreySunwold
My table Looks a bit like this:
MessageID, WebserviceID, SentTime, Status, Message, Error,
Or from a customer/order context if it makes it easer:
OrderID, CustomerID, StatusChangedDate, Status, WidgetName, Comments
Edit 2:
I eventually worked out something
(Hat tip to #StephenChung who basically came up with the exact same, but in classic linq)
var q = myTable.Where(d => d.EndTime > DateTime.Now.AddDays(-1))
.GroupBy(g => g.ConfigID)
.Select(g =>new
{
ConfigID = g.Key,
Data = g.OrderByDescending(d => d.EndTime)
.Take(3).Select(s => new
{
s.Status,
s.SentTime
})
}).ToList();
It does take a while to execute. So I am not sure if this is the most efficient expression.
This should give the last 3 orders of each customer (if having orders at all):
from o in db.Orders
group o by o.CustomerID into g
select new {
CustomerID=g.Key,
LastOrders=g.OrderByDescending(o => o.TimeEntered).Take(3).ToList()
}
However, I suspect this will force the database to return the entire Orders table before picking out the last 3 for each customer. Check the SQL generated.
If you need to optimize, you'll have to manually construct a SQL to only return up to the last 3, then make it into a view.
You can use SelectMany for this purpose:
customers.SelectMany(x=>x.orders.OrderByDescending(y=>y.Date).Take(n)).ToList();
How about this? I know it'll work with regular collections but don't know about EF.
yourCollection.OrderByDescending(item=>item.Date).Take(n);
var ordersByCustomer =
db.Customers.Select(c=>c.Orders.OrderByDescending(o=>o.OrderID).Take(n));
This will return the orders grouped by customer.
var orders = orders.Where(x => x.CustomerID == 1).OrderByDescending(x=>x.Date).Take(4);
This will take last 4 orders. Specific query depends on your table / entity structure.
Btw: You can take x as a order. So you can read it like: Get orders where order.CustomerID is equal to 1, OrderThem by order.Date and take first 4 'rows'.
Somebody might correct me here, but i think doing this is linq with a single query is probably very difficult if not impossible. I would use a store procedure and something like this
select
*
,RANK() OVER (PARTITION BY c.id ORDER BY o.order_time DESC) AS 'RANK'
from
customers c
inner join
order o
on
o.cust_id = c.id
where
RANK < 10 -- this is "n"
I've not used this syntax for a while so it might not be quite right, but if i understand the question then i think this is the best approach.