I'm optimizing my web application and have run into a bottle neck where the SQL generated from my linq expression is very slow.
The following SQL executes in well under a second:
SELECT
ISNULL(COUNT(distinct JOBIDNumber),0),
ISNULL(SUM(JIMQuantityActual * JIMNetMarginFactor),0),
ISNULL(sum((isnull(MATRecoverablePercent,0) / 100) * JIMQuantityActual * JIMNetMarginFactor),0),
ISNULL(sum(CarbonSaving),0)
FROM
dbo.fn_GetJobsForUser(183486) jb
inner join cd_JobMaterial on JIMJobId = jb.JOBIDNumber
WHERE
JOBCollectionDate >= '2014-11-01'
Whereas the sql output by the following query takes between 4 and 16 seconds over the same data:
DateTime d = new DateTime(2014, 11, 1)
from job in sp.sp_GetJobsForUser(183486)
where job.JOBCollectionDate >= d
join material in UnitOfWork.Repository<cd_JobMaterial>().Queryable()
on job.JOBIDNumber equals material.JIMJobId
group material by 1 into f
select new
{
Jobs = f.Distinct().Count(),
Weight = f.Sum(x=> x.JIMQuantityActual * x.JIMNetMarginFactor),
Carbon = f.Sum(x=> x.CarbonSaving),
Recovery = f.Sum(x => ((x.MATRecoverablePercent / 100) * x.JIMQuantityActual * x.JIMNetMarginFactor))
}
Which outputs the following:
-- Region Parameters
DECLARE #contactId Int = 183486
DECLARE #p__linq__0 DateTime2 = '2014-11-01 00:00:00.0000000'
-- EndRegion
SELECT
[Project4].[C1] AS [C1],
[Project4].[C5] AS [C2],
[Project4].[C2] AS [C3],
[Project4].[C3] AS [C4],
[Project4].[C4] AS [C5]
FROM (SELECT
[Project2].[C1] AS [C1],
[Project2].[C2] AS [C2],
[Project2].[C3] AS [C3],
[Project2].[C4] AS [C4],
(SELECT
COUNT(1) AS [A1]
FROM (SELECT DISTINCT
/*Fields omitted for brevity */
FROM [dbo].[fn_GetJobsForUser](#contactId) AS [Extent3]
INNER JOIN (SELECT
/*Fields omitted for brevity */
FROM [dbo].[cd_JobMaterial] AS [cd_JobMaterial]) AS [Extent4]
ON [Extent3].[JOBIDNumber] = [Extent4].[JIMJobId]
WHERE ([Extent3].[JOBCollectionDate] >= #p__linq__0)
AND ([Project2].[C1] = 1)) AS [Distinct1])
AS [C5]
FROM (SELECT
#contactId AS [contactId],
#p__linq__0 AS [p__linq__0],
[GroupBy1].[K1] AS [C1],
[GroupBy1].[A1] AS [C2],
[GroupBy1].[A2] AS [C3],
[GroupBy1].[A3] AS [C4]
FROM (SELECT
[Project1].[K1] AS [K1],
SUM([Project1].[A1]) AS [A1],
SUM([Project1].[A2]) AS [A2],
SUM([Project1].[A3]) AS [A3]
FROM (SELECT
1 AS [K1],
[Project1].[JIMQuantityActual] * [Project1].[JIMNetMarginFactor] AS [A1],
[Project1].[CarbonSaving] AS [A2],
([Project1].[MATRecoverablePercent] / CAST(100 AS DECIMAL(18))) * [Project1].[JIMQuantityActual] * [Project1].[JIMNetMarginFactor] AS [A3]
FROM (SELECT
[Extent2].[MATRecoverablePercent] AS [MATRecoverablePercent],
[Extent2].[JIMQuantityActual] AS [JIMQuantityActual],
[Extent2].[JIMNetMarginFactor] AS [JIMNetMarginFactor],
[Extent2].[CarbonSaving] AS [CarbonSaving]
FROM [dbo].[fn_GetJobsForUser](#contactId) AS [Extent1]
INNER JOIN (SELECT
/*Fields omitted for brevity */
FROM [dbo].[cd_JobMaterial] AS [cd_JobMaterial]) AS [Extent2]
ON [Extent1].[JOBIDNumber] = [Extent2].[JIMJobId]
WHERE [Extent1].[JOBCollectionDate] >= #p__linq__0) AS [Project1]) AS [Project1]
GROUP BY [K1]) AS [GroupBy1]) AS [Project2]) AS [Project4]
How do I re-write the linq expression to produce more efficient sql or is it just a case of writing a stored procedure and using that instead?
Unfortunately that's one of the downsides of using something as flexible as Entity Framework that has to support a wide variety of complex translations. It obviously comes with great benefit in other areas though, so you'll have to balance those with the performance aspect.
Even if you could find a way to rewrite the query that would generate better SQL now, that's subject to changes in the underlying provider in future versions. Enjoy the clean, concise code for as long as you can before performance is no longer acceptable for your application. If EF isn't cutting it at that point, then use some of the hooks provided to execute raw SQL, stored procedures, etc. that won't be as pretty.
Related
SELECT P.OBJECT_NAME||'.'||P.PROCEDURE_NAME,P.OBJECT_ID, R.IN_OUT,COUNT(*)
FROM ALL_PROCEDURES#ORCL P, ALL_ARGUMENTS#ORCL R
WHERE P.OWNER = 'SWPROEAIDB3'
AND P.OBJECT_TYPE = 'PACKAGE'
AND P.PROCEDURE_NAME IS NOT NULL
AND P.OBJECT_ID=R.OBJECT_ID
AND P.OBJECT_ID IN (SELECT OBJECT_ID
FROM ALL_ARGUMENTS#ORCL
WHERE OWNER='SWPROEAIDB3'
GROUP BY OBJECT_ID
HAVING COUNT(*)=2 )
GROUP BY P.OBJECT_NAME||'.'||P.PROCEDURE_NAME,
P.OBJECT_ID,
R.IN_OUT
HAVING COUNT(*)=1
I am looking for all such procedures for the schema - SWPROEAIDB3 who have only 2 arguments in them and must contain one IN and one OUT.
We have a query run by our development team that's heavy in resources and looking at the explain plan, it looks like its uses the same data set multiple times. Is there anyway we can re-write this query.
Now, i tried to replace the co-related query with direct join but still the multiple co-related queries look the same apart from one minor difference.
select tb2.mktg_id, mktg_cd , count(distinct tb2.conf_id)
from
(select conf_id, count(distinct c.mktg_id) as num_cpg
from acc_latst c, off_latst ot
where c.mktg_id = ot.mktg_id and c.bus_eff_dt > '2019-01-01' and to_date(strt_tms) = '2019-01-10'
group by conf_id
having count(distinct c.mktg_id) >1
)tb1,
(select distinct conf_id, c.mktg_id, mktg_cd
from acc_latst c, off_latst ot
where c.mktg_id = ot.mktg_id and c.bus_eff_dt > '2019-01-01' and to_date(strt_tms) = '2019-01-10'
)tb2
where tb1.conf_id = tb2.conf_id group by tb2.mktg_id, mktg_cd
One way is using CTE -
with res1 as
(
select distinct conf_id, c.mktg_id, mktg_cd
from acc_latst c, off_latst ot
where c.mktg_id = ot.mktg_id and c.bus_eff_dt > '2019-01-01' and to_date(strt_tms) = '2019-01-10'
)
,res2 as
(
select conf_id, count(distinct c.mktg_id) as num_cpg
from res1 group by conf_id having count(distinct c.mktg_id) > 1
)
select res1.mktg_id, mktg_cd, count(distinct res1.conf_id) from res1 t1 inner join res2 t2 on t1.conf_id=t2.conf_id group by res1.mktg_id, mktg_cd;
If the query is still slow, could you provide table and partition details.
I use Yii2 a lot but normally with MySQL. This is my first time using it with Oracle. I am finding the performance quite poor though. I've done some googling around and see that although it doesn't appear that common to use Yii2 + Oracle there does seem to be people talking about it being quite slow.
I'm wondering what advice/tips/processes/packages can be employed to improve performance?
I am loading a very small table. It has 8 rows, the entire table has 4 columns. They're are 27 DB calls using 6.4MB of data, that takes about 3 seconds to run. Looking at the logs I see a lot of stuff I don't really understand:
SELECT D.CONSTRAINT_NAME, D.CONSTRAINT_TYPE, C.COLUMN_NAME, C.POSITION, D.R_CONSTRAINT_NAME, E.TABLE_NAME AS TABLE_REF, F.COLUMN_NAME AS COLUMN_REF, C.TABLE_NAME
FROM
ALL_CONS_COLUMNS C
INNER JOIN ALL_CONSTRAINTS D ON D.OWNER = C.OWNER AND D.CONSTRAINT_NAME = C.CONSTRAINT_NAME
LEFT JOIN ALL_CONSTRAINTS E ON E.OWNER = D.R_OWNER AND E.CONSTRAINT_NAME = D.R_CONSTRAINT_NAME
LEFT JOIN ALL_CONS_COLUMNS F ON F.OWNER = E.OWNER AND F.CONSTRAINT_NAME = E.CONSTRAINT_NAME AND F.POSITION = C.POSITION
and
SELECT a.column_name, a.data_type, a.data_precision, a.data_scale, a.data_length,
a.nullable, a.data_default,
( SELECT D.constraint_type
FROM ALL_CONS_COLUMNS C
inner join ALL_constraints D on D.OWNER = C.OWNER and D.constraint_name = C.constraint_name
WHERE C.OWNER = B.OWNER
and C.table_name = B.object_name
and C.column_name = A.column_name
and D.constraint_type = 'P') as Key,
com.comments as column_comment
FROM ALL_TAB_COLUMNS A
inner join ALL_OBJECTS B ON b.owner = a.owner and ltrim(B.OBJECT_NAME) = ltrim(A.TABLE_NAME)
LEFT JOIN all_col_comments com ON (A.owner = com.owner AND A.table_name = com.table_name AND A.column_name = com.column_name)
Then more stuff about triggers.
I also get errors when I try to use relations such as
Model::find()->with('relationName')->all();
I get
'ORA-01795: maximum number of expressions in a list is 1000 error.
These tables collectively have about 17k rows. There's about 11k in one table and about 7k in the other that are linked.
I'm using fk constraints as references within the tables.
Add/Use following settings in your database configurations.
'schemaCacheDuration' => 7200,
'schemaCache' => 'cache',
'enableSchemaCache' => true,
And
change YII_DEBUG from true to false and YII_ENV from dev to prod.
This will reduce unnecessary SQL executions.
This query is not showing anything at all. I want results only those ORDERNO which match in the sub-query.
SELECT SUM(STY.NQTY * STY.NUNITPRICE
/ 1000) AS ORDERAMOUNT,
(SELECT SUM(TBLORDERSTYLE.NQTY * TBLORDERSTYLE.NUNITPRICE/1000) FROM TBLORDER
INNER JOIN TBLORDERSTYLE
ON TBLORDER.CORDERNO = TBLORDERSTYLE.CORDERNO
WHERE TBLORDER.CLCNO LIKE '%MR#%' AND
TBLORDER.CORDERNO = ORD.CORDERNO
GROUP BY ORD.CORDERNO) AS K
FROM TBLORDER ORD INNER JOIN
TBLORDERSTYLE STY ON
ORD.CORDERNO = STY.CORDERNO
where (ORD.NPOSTFLAG = '1') AND
(ORD.NCANCEL = '0') AND
ORD.NPAYMODE = '1' AND
(ORD.DPICONFIRMDATE BETWEEN
TO_DATE('01-JUL-2012')AND TO_DATE('31-JUL-2012'))
The error message is quite clear: you don't have a GROUP BY clause. Whenever we mix aggregating functions like SUM() with non-aggregated columns we need to include the static columns in a GROUP BY clause.
In you code the static column is K. Yes it is derived from an aggregation but that is in a sub-query, and so it counts as a non-aggregating column.
The way to solve thi sis to move the sub-query into a common table expression (what Oracle calls subquery factoring), which makes it easy to reference the column more than once. The use of the WITH cluase is covered in the Oracle SQL Reference. Find out more.
with cte as (SELECT ORD.CORDERNO
, SUM(TBLORDERSTYLE.NQTY * TBLORDERSTYLE.NUNITPRICE/1000) as k
FROM TBLORDER
INNER JOIN TBLORDERSTYLE
ON TBLORDER.CORDERNO = TBLORDERSTYLE.CORDERNO
WHERE TBLORDER.CLCNO LIKE '%MR#%' AND
TBLORDER.CORDERNO = ORD.CORDERNO
GROUP BY ORD.CORDERNO)
SELECT SUM(STY.NQTY * STY.NUNITPRICE
/ 1000) AS ORDERAMOUNT,
cte.K
FROM cte inner join
TBLORDER ORD on ord.orderno = k.orderno
INNER JOIN TBLORDERSTYLE STY ON STY.CORDERNO = k.orderno
where (ORD.NPOSTFLAG = '1') AND
(ORD.NCANCEL = '0') AND
ORD.NPAYMODE = '1' AND
(ORD.DPICONFIRMDATE BETWEEN TO_DATE('01-JUL-2012')AND TO_DATE('31-JUL-2012'))
group by cte.k
It's highly likely that this is not the actual logic you need. You did not include any explanation of the business rules you're trying to implement so I have just made a guess at how to join the sub-query with the main query.
SQL Query (Execution plan cost = 0.0127553)
SELECT
SUM(DATEDIFF(second, DTActivate, DTDeActivate)) AS Seconds,
AA.ID AS AAID,
AA.WorkStation
FROM
DbLogItems I
INNER JOIN DbApplicationArguments AA ON
AA.Id = I.ApplicationArgument_ID
GROUP BY
AA.ID,
AA.WorkStation
C#
var q = from items in db.LogItem
join aa in db.ApplicationArguments on
items.ApplicationArgument.ID equals aa.ID
into aaGroup
from aaJoin in aaGroup.DefaultIfEmpty()
group items by new
{
aaJoin.ID,
aaJoin.WorkStation
} into grouping
select new
{
Seconds = grouping.Sum(x => SqlFunctions.DateDiff("second", x.DTActivate, x.DTDeActivate)),
grouping.Key.ID,
grouping.Key.WorkStation
};
Result SQL very big (Execution plan cost = 0.0199849)
SELECT
1 AS [C1],
[GroupBy1].[A1] AS [C2],
[GroupBy1].[K1] AS [ID],
[GroupBy1].[K2] AS [WorkStation]
FROM
(
SELECT
[Join1].[K1] AS [K1],
[Join1].[K2] AS [K2],
SUM([Join1].[A1]) AS [A1]
FROM
(
SELECT
[Extent2].[ID] AS [K1],
[Extent2].[WorkStation] AS [K2],
DATEDIFF(second, [Extent1].[DTActivate], [Extent1].[DTDeActivate]) AS [A1]
FROM
[dbo].[DbLogItems] AS [Extent1]
LEFT OUTER JOIN [dbo].[DbApplicationArguments] AS [Extent2]
ON [Extent1].[ApplicationArgument_ID] = [Extent2].[ID]
) AS [Join1]
GROUP BY
[K1],
[K2]
) AS [GroupBy1]
help please write correct linq code.
My SQL Execution plan cost = 0.0127553.
Linq SQL Execution plan cost = 0.0199849.
DIFF = 0,0072296 on 21+10 records
LINQ to SQL queries are what they are. Sometimes you can't emit better SQL. However, you can write plain old T-SQL and call it in a number ways: a table-valued UDF perhaps that your customized DataContext exposes as an IQueryable.