There is this stored procedure that builds a dynamic query string and then execute it. The sp works fine in development and testing environment, but the DBA of the client company has informed that this query is hitting really hard to the database in production. The IT area has asked us to tune up the query. So far so good, we've moved almost all this sp from building the query string dynamically into a single big query that performs really fast (compared to the old query).
We have found (among other things) that the sp built the where clause of the query string by evaluating if a parameter has a default value or a real value i.e.
IF P_WORKFLOWSTATUS <> 0 THEN
L_SQL := TRIM(L_SQL) || ' AND WORKFLOW.STATUS = ' || TO_CHAR(P_WORKFLOWSTATUS);
END IF;
So we optimized this behavior to
WHERE
...
AND (WORKFLOW.STATUS = P_WORKFLOWSTATUS OR P_WORKFLOWSTATUS = 0)
This kind of change has improved the query that affected numeric columns, but we have found a problem with a VARCHAR2 parameter and column. The current behavior is
--CLIENT.CODE is a VARCHAR2(14) column and there is an unique index for this column.
--The data stored in this field is like 'N0002077123', 'E0006015987' and similar
IF NVL(P_CLIENT_CODE, '') <> '' THEN
L_SQL := TRIM(L_SQL) || ' AND CLIENT.CODE = ''' || P_CLIENT_CODE || '''';
END IF;
We tried to change this to our optimized version of the query by doing
WHERE
...
AND (CLIENT.CODE = P_CLIENT_CODE OR NVL(P_CLIENT_CODE, '') = '')
but this change made the query lost performance. Is there a way to optimize this part of the query or should we turn our big query into a dynamic query (again) just to evaluate if this VARCHAR2 parameter should be added or not into the where clause?
Thanks in advance.
Oracle treats empty strings '' as NULL. So this condition NVL(P_CLIENT_CODE, '') = '' doesn't really make much sense. Moreover it will always be false, because here we are checking equality of NULLs, which is always false. To that end you might and probably should recode that part of the query as:
WHERE
...
AND ( (CLIENT.CODE = P_CLIENT_CODE) OR (CLIENT IS NULL) )
I recommend or to move this varchar2 parameters back to dynamic, or to use the following:
WHERE
...
AND CLIENT.CODE = nvl(P_CLIENT_CODE,CLIENT.CODE)
and be sure you have index on client.code.(Or the table partitioned on client.code, if possible.)
Of course, as it has already been said, you need need to perform correct null checks.
However, the trick is, the difference between
AND (CLIENT.CODE = P_CLIENT_CODE OR NVL(P_CLIENT_CODE, '') = '')
and
AND ( (CLIENT.CODE = P_CLIENT_CODE) OR (CLIENT IS NULL) )
is very unlikely to cause performance problems only by itself. I would even say that query with second clause could perform worse than with the first one, as it will yield true for more rows, resulting in larger result set for consequent joins/orders/filers etc.
I'd bet that adding this clause to your query somehow breaks its optimal execution plan. For instance, having obsolete statistics, the optimizer could make a sub-optimal decision to choose unselective index on client.code instead of others available.
However, it is hard to tell for sure without seeing actual (not the expected one, which you obtain with explain plan command!) execution plan of slow query and your table structure.
Related
I am new to Oracle apex.
I have a page with a form that is used to enter data into a table.
For Example there is P_ID, P_NAME, P_ADD_USER, P_VERIFIED_USER, P_SECOND_ID items. I want to make P_SECOND_ID read only based on multiple conditions.
Condition is
IF P_ADD_USER <> :APP_USER AND P_SECOND_ID = ' ' THEN
'P_SECOND_ID should be available to Edit.
ELSE
P_SECOND_ID will be read only.
I tried to use Type = Item!=Value but it is allowing me to add only one condition so is there any option that i can use both conditions and make that ITEM read only.
Condition I'd suggest in such a case is a function that returns a Boolean - if it returns TRUE, something will happen; otherwise, it won't.
So, if you want to make P_SECOND_ID editable if conditions you mentioned are satisfied, then you'd
return not ( :P_ADD_USER <> :APP_USER
and :P_SECOND_ID = ' '
);
Though, did you really want to say :P_SECOND_ID = ' '? Is there a space character in it? Should that, perhaps, be :P_SECOND_ID IS NULL?
SQL Expression allows you to include multiple conditions. You might want to rephrase your requirement to make the item read only ( versus making it editable ) under certain conditions. If I get your requirement right, the condition to make P_SECOND_ID read only is when
:P_ADD_USER = :APP_USER OR :P_SECOND_ID is not NULL
You can use this expression directly in the SQL Expression.
I've gone down a bit of a path and hit a wall with how this could be possibly achieved.
Basically, a query is constructed using JPA and passed to an Oracle DB. On the DB there is a Package, used to generate a reference, and this is dynamically named, based on the environment. This value is user-editable, and stored as a DB property within the application. I don't have any control over the architecture of this.
At a pre-JPA stage, a Query String is generated using the reference value for the Package, which is set as a property (again, I can't change the way this has been designed). I set this up using the Query method setParameter(), like so:
(pseudocode replacing the irrelevant parts for focused context)
String referenceRef = [ reference is fetched from DB properties ];
String queryString = "SELECT ?1 FROM sys.dual";
final Query myQuery = getEntityManager().createNativeQuery( queryString );
myQuery.setParameter( 1, referenceRef );
return myQuery.getSingleResult();
I pretty much did this as a reflex, only to realise (in retrospec, quite obviously) that this won't actually work, as it is escaping the element that should not be escaped...
So, where the referenceRef = "DynamicallyNamedPackage.DoThisDynamicallyNamedThing", the above code will just return "DynamicallyNamedPackage.DoThisDynamicallyNamedThing", as it is obviously making it safe, and the point of doing so is, to a certain extent, the antethesis of what I'm trying to do.
Is it possible to achieve this without creating a whole chunk of additional code? All I can currently think of, as an alternative, is to query dba_procedures for all package objects that match, and using the result of that query to construct the queryString (hence circumnavigating using any user-editable values), but it feels like it's going to be convoluted. This is the alternative, which I am using in lieu of an improvement:
final String verifyReference = "SELECT object_name FROM "
+ "dba_procedures WHERE object_type = 'PACKAGE' AND object_name =?1";
final Query refQuery = getEntityManager().createNativeQuery( verifyReference );
refQuery.setParameter( 1, referenceRef );
final String result = refQuery.getSingleResult();
final String queryString = "SELECT " + result + " FROM sys.dual";
final Query myQuery = getEntityManager().createNativeQuery( queryString );
return myQuery.getSingleResult();
It will essentially look up the user-editable property reference against a list of existing packages, then use the result of that query for building the original reference. It has more null checking and so on involved, and does remove the vulnerability, but feels a bit 'unpolished'.
(As has already been mentioned in the comments, this sort of is designed to need a SQL injection, but needs to prevent "SQL Injection" as a definition of not allowing the DB to be manipulated outside of the design by using an unintended value.)
The Oracle dictionary view all_procedures contains a list of all procedures accessible to the current user.
Specifically in the view there are columns OWNER, OBJECT_NAME (=package name), PROCEDURE_NAME.
You may use this view to sanitize the configured input by simple adding an EXISTS subquery such as:
select
?
from dual where exists (
select null from all_procedures
where
OWNER||'.'||OBJECT_NAME||'.'||PROCEDURE_NAME = upper(?) and
object_type = 'PACKAGE');
You will have to bind twice the same input parameter.
The query returns no data if there is not procedure with the given name, so you may raise an exception.
The query above expects a full qualified stored procedure name, i.e. owner.package.procedure, you'll have to adapt it slightly if you allow unqualified names (without the owner).
I have 2 values in table User: Address1, Address2. Both could be null. As part of a filter method, I am attempting something like the below:
var tempUsers = users.Where(q => q.Address1.ToLower().Contains(address.ToLower()) || q.Address2.ToLower().Contains(address.ToLower()));
This is returning a Null Reference Exception, and rightly so.
Linq queries need to be handled against null values
I would be attempting
null.ToLower() and null.Contains() within the query
What is the best way to go around it? If it was a simple 1 field Query, for e.g. just Address1, I would have simply filtered out all items with empty Address1, and continued normally in the second query. In this case, both fields are important to the filtering, as in, the input: address could be either in Address1 or Address2 of the User table.
I know this might not be possible in a 1 liner, but what is the best approach to take in terms of time and performance?
How about this:
var address = (GetAddressFromOuterWorld() ?? String.Empty).ToLower();
var tempUsers = users.Where(user => (user.Address1 ?? String.Empty).ToLower().Contains(address)
|| (user.Address2 ?? String.Empty).ToLower().Contains(address));
This definitely works with LINQ to Object, but probably fails with LINQ to SQL, but in that case you normally write user.Address1 == address || user.Addrss2 == address and your database uses a case-insensitive collate setting.
You can easily add null checks like this.
var tempUsers = users.Where(q =>
(!string.IsNullOrEmpty(q.Address1) && q.Address1.ToLower().Contains(address.ToLower())) ||
(!string.IsNullOrEmpty(q.Address2) && q.Address2.ToLower().Contains(address.ToLower())));
I started using compiled queries to increase the performance of some commonly executed linq to entities queries. In one scenario I only boiled the query down to it's most basic form and pre-compiled that, then I tack on additional where clauses based on user input.
I seem to be losing the performance benefit of compiled queries in this particular case. Can someone explain why?
Here's an example of what I'm doing...
IEnumerable<Task> tasks = compiledQuery.Invoke(context, userId);
if(status != null)
{
tasks = tasks.Where(x=x.Status == status);
}
if(category != null)
{
tasks = tasks.Where(x=x.Category == category);
}
return tasks;
I think it's important to understand how Compiled Queries in EF work.
When you execute a query Entity Framework will map your expression tree with the help of your mapping file (EDMX or with code first your model definitions) to a SQL query. This can be a complex and performance intensive task.
Precompiling stores the results of these mapping phase so the next time you hit the query it has the SQL already available and it only has to set the current parameters.
The problem is that a precompiled query will lose it's performance benefit as soon as you modifie the query. Let's say you have the following:
IQueryable query = GetCompiledQuery(); // => db.Tasks.Where(t => t.Id == myId);
var notModifiedResult = query.ToList(); // Fast
int ModifiedResult = query.Count(); // Slow
With the first query you will have all the benefits of precompiling because EF has the SQL already generated for you and can execute this immediatly.
The second query will lose the precompiling because it has to regenerate it's SQL.
If you would now execute a query on notModifiedResult this will be a Linq To Objects one because you have already executed your SQL to the database and fetched all the elements in memory.
You can however chain Compiled Queries (that is, use a compiled query in another compiled query).
But your code would require a series of compiled queries:
- The default
- One where status != null
- One where category != null
- One where both status and category != null
(Note: I haven't done any EF work for ages, and then it was just pottering. This is just an informed guess, really.)
This could be the culprit:
IEnumerable<Task> tasks = compiledQuery.Invoke(context, userId);
Any further querying will have to be done within the .NET process, not in SQL. All the possible results will have to be fetched from the database and filtered locally. Try this instead:
IQueryable<Task> tasks = compiledQuery.Invoke(context, userId);
(Assuming that's valid, of course.)
The compiled query can't be changed, only the parameters can be changed. What you are doing here is actually running the query, and THEN filtering the results.
.Invoke(context, userId); // returns all the results
.Where(....) // filters on that entire collection
You can see if there is a clever way to restate your query, so that the parameters can be included in all cases, but not have any effect. I haven't worked with compiled queries, sorry about that, but does this work (using -1 as the "ignore" value)?
// bunch of code to define the compiled query part, copied from [msdn][1]
(ctx, total) => from order in ctx.SalesOrderHeaders
where (total == -1 || order.TotalDue >= total)
select order);
In SQL, you do this by either using dynamic sql, or having a default value (or null) that you pass in which indicates that parameter should be ignored
select * from table t
where
(#age = 0 or t.age = #age) and
(#weight is null or t.weight = #weight)
I'm trying to do a query like so...
query.Where(x => !string.IsNullOrEmpty(x.PropertyName));
but it fails...
so for now I have implemented the following, which works...
query.Where(x => (x.PropertyName ?? string.Empty) != string.Empty);
is there a better (more native?) way that LINQ handles this?
EDIT
apologize! didn't include the provider... This is using LINQ to SQL
http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=367077
Problem Statement
It's possible to write LINQ to SQL that gets all rows that have either null or an empty string in a given field, but it's not possible to use string.IsNullOrEmpty to do it, even though many other string methods map to LINQ to SQL.
Proposed Solution
Allow string.IsNullOrEmpty in a LINQ to SQL where clause so that these two queries have the same result:
var fieldNullOrEmpty =
from item in db.SomeTable
where item.SomeField == null || item.SomeField.Equals(string.Empty)
select item;
var fieldNullOrEmpty2 =
from item in db.SomeTable
where string.IsNullOrEmpty(item.SomeField)
select item;
Other Reading:
1. DevArt
2. Dervalp.com
3. StackOverflow Post
This won't fail on Linq2Objects, but it will fail for Linq2SQL, so I am assuming that you are talking about the SQL provider or something similar.
The reason has to do with the way that the SQL provider handles your lambda expression. It doesn't take it as a function Func<P,T>, but an expression Expression<Func<P,T>>. It takes that expression tree and translates it so an actual SQL statement, which it sends off to the server.
The translator knows how to handle basic operators, but it doesn't know how to handle methods on objects. It doesn't know that IsNullOrEmpty(x) translates to return x == null || x == string.empty. That has to be done explicitly for the translation to SQL to take place.
This will work fine with Linq to Objects. However, some LINQ providers have difficulty running CLR methods as part of the query. This is expecially true of some database providers.
The problem is that the DB providers try to move and compile the LINQ query as a database query, to prevent pulling all of the objects across the wire. This is a good thing, but does occasionally restrict the flexibility in your predicates.
Unfortunately, without checking the provider documentation, it's difficult to always know exactly what will or will not be supported directly in the provider. It looks like your provider allows comparisons, but not the string check. I'd guess that, in your case, this is probably about as good of an approach as you can get. (It's really not that different from the IsNullOrEmpty check, other than creating the "string.Empty" instance for comparison, but that's minor.)
... 12 years ago :) But still, some one may found it helpful:
Often it is good to check white spaces too
query.Where(x => !string.IsNullOrWhiteSpace(x.PropertyName));
it will converted to sql as:
WHERE [x].[PropertyName] IS NOT NULL AND ((LTRIM(RTRIM([x].[PropertyName])) <> N'') OR [x].[PropertyName] IS NULL)
or other way:
query.Where(x => string.Compare(x.PropertyName," ") > 0);
will be converted to sql as:
WHERE [x].[PropertyName] > N' '
If you want to go change the type of the collection from nullable type IEnumerable<T?> to non-null type IEnumerable<T> you can use .OfType<T>().
.OfType<T>() will remove null values and return a list of the type T.
Example: If you have a list of nullable strings: List<string?> you can change the type of the list to string by using OfType<string() as in the below example:
List<string?> nullableStrings = new List<string?> { "test1", null, "test2" };
List<string> strings = nullableStrings.OfType<string>().ToList();
// strings now only contains { "test1", "test2" }
This will result in a list of strings only containing test1 and test2.