I'm tasked with adding an option to our search, which will return results where a given field doesn't begin with a letter of the alphabet. (The .StartsWith(letter) part wasn't so hard).
But I'm rather unsure about how to get the results that don't fall within the A-Z set, and equally hoping it generates some moderately efficient SQL underneath.
Any help appreciated - thanks.
In C# use the following construct, assuming db as a data context:
var query = from row in db.SomeTable
where !System.Data.Linq.SqlClient.SqlMethods.Like(row.SomeField, "[A-Z]%")
select row;
This is only supported in LINQ to SQL queries. All rules of the T-SQL LIKE operator apply.
You could also use less effective solution:
var query = from row in db.SomeTable
where row.SomeField[0] < 'A' || row.SomeField[0] > 'Z'
select row;
This gets translated into SUBSTRING, CAST, and UNICODE constructs.
Finally, you could use VB, where there appears to be a native support for the Like method.
Though SQL provides the ability to check a range of characters in a LIKE statement using bracket notation ([a-f]% for example), I haven't seen a linq to sql construct that supports this directly.
A couple thoughts:
First, if the result set is relatively small, you could do a .ToList() and filter in memory after the fact.
Alternatively, if you have the ability to change the data model, you could set up additional fields or tables to help index the data and improve the search.
--EDIT--
Made changes per Ruslan's comment below.
Well, I have no idea if this will work because I have never tried it and don't have a compiler nearby to try it, but the first thing I would try is
var query = from x in db.SomeTable
where x.SomeField != null &&
x.SomeField.Length >= 1 &&
x.SomeField.Substring(0, 1).All(c => !Char.IsLetter(c))
select x;
The possiblility exists that LINQ to SQL fails to convert this to SQL.
Related
I have the following LINQ query; I want to remove dot character if available at the end of string
var hatoList = (from L in db.vHatoList select L.HatoFullName.Trim('.')).ToList();
You're currently trying to perform the query in the database. We don't know what database technology you're using, so it's hard to know whether there's some way to make it actually happen in the database, but it's probably simplest to do in memory instead (given that it won't change how many records will be returned). The AsEnumerable() method is used to just change the compile-time type to IEnumerable<T> (instead of IQueryable<T>) forcing the rest of the query to be evaluated in .NET code instead of in the database:
var hatoList = db.vHatoList
.AsEnumerable()
.Select(entry => entry.HatoFullName.TrimEnd('.'))
.ToList();
(I've also changed the call from Trim to TrimEnd to avoid it removing periods at the start of the string.)
You are using linq to sql and database does not understand TrimEnd or Trim.
So first you have need to load data in memory and then use TrimEnd method.
var hatoList = (from L in db.vHatoList select L.HatoFullName).ToList().
Select(t=>t.TrimEnd('.'));
I have below query, because of the huge data in the MATTER Table, it is taking huge time for LIKE statement to execute, so I was thinking of using the CONTEXT Index and using CONTAIN.
Shall I do indexing only on Matter_title or some other column as well,. Based on the below select query
Inputs highly appreciated
SELECT DISTINCT dm.MATTER_SEQ
FROM MATTER dm
,MATTER_TYPE dmt
,MATTER_SUBTYPE dms
,STATUS ds
,FILING df
WHERE dm.MATTER_TYPE_SEQ=dmt.MATTER_TYPE_SEQ
AND dm.MATTER_SUBTYPE_SEQ=dms.MATTER_SUBTYPE_SEQ
AND dm.STATUS_CODE NOT IN ('abc','jkl','xyz')
AND dm.STATUS_CODE = DS.STATUS_CODE
AND dm.IS_EXTERNAL='1'
AND dm.IS_DELETED='0'
AND dm.MATTER_SEQ = df.MATTER_SEQ
AND trunc(dm.CREATED_DATE) between '01-NOV-95' AND '02-OCT-18'
AND upper(dm.MATTER_TITLE) like(upper (q'{%jdasuidhajsndjahs%}'))
It sounds like you're already aware that LIKE with a leading wildcard ('%ABC') is notoriously inefficient since it typically can't use indexes and does a full table scan.
If the other optimizing suggestions don't help much, you probably would see better performance with a Context index. Be sure to set the SUBSTRING_INDEX preference so it'll specifically prepare the index for infix searches like yours. See this Ask Tom for more details. (If you will also have wildcards in the middle of strings ('ABC%DEF'), you might also want to set the PREFIX options.)
begin
ctx_ddl.create_preference('SUBSTRING_PREF','BASIC_WORDLIST');
ctx_ddl.set_attribute('SUBSTRING_PREF','SUBSTRING_INDEX','TRUE');
end;
create index matter_title_idx on MATTER(MATTER_TITLE)
indextype is ctxsys.context
parameters ('wordlist SUBSTRING_PREF');
Also note that Context indexes are case-insensitive by default, so you don't need to do UPPER(). I haven't tried using q'' literals with contains, so I'm not sure how this'll work.
AND CONTAINS(dm.MATTER_TITLE, q'{%jdasuidhajsndjahs%}') > 0
Try creating function Indexes upper(dm.MATTER_TITLE) and second trunc(dm.CREATED_DATE).
Also I am considering that the columns in the Join conditions already have indexes. If not have them indexed.
AX allows you to enter basic SQL into View ranges. For example, in an AOT view's range, for the match value, you could enter (StatRepInterval.Name == 'Weekly'). This works nicely.
However, I need to do a more advanced lookup on a View, using a subquery. Can anyone suggest a way to do this?
This is what I would like to use, but I receive an error: "Query extended range failure: Syntax error near 34."
(StatRepInterval.Name == (SELECT FIRSTONLY StatRepInterval.Name FROM StatRepInterval WHERE StatRepInterval.PrintDirection == 1 ORDER BY StatRepInterval.Name DESC))
I've tried a lot of different variants of the subquery, from straight T-SQL to X++ SQL, but nothing seems to work.
Thanks for the help.
Sub-queries are not supported in query expressions.
This may be solved by using additional datasources with inner or outer joins as you observed.
See the spec and Axaptapedida on query expressions.
I found a way to do this. It isn't pretty, and I'm going to leave the question unanswered for a bit, should someone else have a more graceful solution.
Create a source View that contains all fields I wish to return, plus calculated fields that contain my subquery results.
Create a second View that uses the first as a data source, and applies all the necessary ranges.
Works pretty nicely.
Probably inefficient if there were large tables of data, but this is in a relatively small section of AX.
I'm having a problem joining two LINQ queries.
Currently, my (original) code looks like this
s.AnimalTypes.Sort((x, y) => string.Compare(x.Type, y.Type));
What I'm needing to do is change this based on a date, then select all data past that date, so I have
s.AnimalTypes.Select(t=>t.DateChanged > dateIn).ToList()
s.AnimalTypes.Sort((…
This doesn't look right as it's not sorting the data selected, rather sorting everything in s.AnimalTypes.
Is there a way to concatenate the two LINQ lines? I've tried
s.AnimalTypes.Select(t=>t.DateChanged > dateIn).ToList().Sort((…
but this gives me an error on the Sort section.
Is there a simple way to do this? I've looked around and Grouo and OrderBy keep cropping up, but I'm not sure these are what I need here.
Thanks
From your description, I believe you want something like:
var results = s.AnimalTypes.Where(t => t.DateChanged > dateIn).OrderBy(t => t.Type);
You can call ToList() to convert to a List<T> at the end if required.
There are a couple of fundamental concepts I believe you are missing here -
First, unlike List<T>.Sort, the LINQ extension methods don't change the original collections, but rather return a new IEnumerable<T> with the filtered or sorted results. This means you always need to assign something to the return value (hence my var results = above).
Second, Select performs a mapping operation - transforming the data from one form to another. For example, you could use it to extract out the DateChanged (Select(t => t.DateChanged)), but this would give you an enumeration of dates, not the original animal types. In order to filter or restrict the list with a predicate (criteria), you'd use Where instead.
Finally, you can use OrderBy to reorder the resulting enumerable.
You are using Select when you actually want to use Where.
Select is a projection from one a collection of one type into another type - you won't increase or reduce the number of items in a collection using Select, but you can instead select each object's name or some other property.
Where is what you would use to filter a collection based on a boolean predicate.
The Situation
I have some trouble with my query execution plan for a medium-sized query over a large amount of data in Oracle 11.2.0.2.0. In order to speed things up, I introduced a range filter that does roughly something like this:
PROCEDURE DO_STUFF(
org_from VARCHAR2 := NULL,
org_to VARCHAR2 := NULL)
-- [...]
JOIN organisations org
ON (cust.org_id = org.id
AND ((org_from IS NULL) OR (org_from <= org.no))
AND ((org_to IS NULL) OR (org_to >= org.no)))
-- [...]
As you can see, I want to restrict the JOIN of organisations using an optional range of organisation numbers. Client code can call DO_STUFF with (supposed to be fast) or without (very slow) the restriction.
The Trouble
The trouble is, PL/SQL will create bind variables for the above org_from and org_to parameters, which is what I would expect in most cases:
-- [...]
JOIN organisations org
ON (cust.org_id = org.id
AND ((:B1 IS NULL) OR (:B1 <= org.no))
AND ((:B2 IS NULL) OR (:B2 >= org.no)))
-- [...]
The Workaround
Only in this case, I measured the query execution plan to be a lot better when I just inline the values, i.e. when the query executed by Oracle is actually something like
-- [...]
JOIN organisations org
ON (cust.org_id = org.id
AND ((10 IS NULL) OR (10 <= org.no))
AND ((20 IS NULL) OR (20 >= org.no)))
-- [...]
By "a lot", I mean 5-10x faster. Note that the query is executed very rarely, i.e. once a month. So I don't need to cache the execution plan.
My questions
How can I inline values in PL/SQL? I know about EXECUTE IMMEDIATE, but I would prefer to have PL/SQL compile my query, and not do string concatenation.
Did I just measure something that happened by coincidence or can I assume that inlining variables is indeed better (in this case)? The reason why I ask is because I think that bind variables force Oracle to devise a general execution plan, whereas inlined values would allow for analysing very specific column and index statistics. So I can imagine that this is not just a coincidence.
Am I missing something? Maybe there is an entirely other way to achieve query execution plan improvement, other than variable inlining (note I have tried quite a few hints as well but I'm not an expert on that field)?
In one of your comments you said:
"Also I checked various bind values.
With bind variables I get some FULL
TABLE SCANS, whereas with hard-coded
values, the plan looks a lot better."
There are two paths. If you pass in NULL for the parameters then you are selecting all records. Under those circumstances a Full Table Scan is the most efficient way of retrieving data. If you pass in values then indexed reads may be more efficient, because you're only selecting a small subset of the information.
When you formulate the query using bind variables the optimizer has to take a decision: should it presume that most of the time you'll pass in values or that you'll pass in nulls? Difficult. So look at it another way: is it more inefficient to do a full table scan when you only need to select a sub-set of records, or to do indexed reads when you need to select all records?
It seems as though the optimizer has plumped for full table scans as being the least inefficient operation to cover all eventualities.
Whereas when you hard code the values the Optimizer knows immediately that 10 IS NULL evaluates to FALSE, and so it can weigh the merits of using indexed reads for find the desired sub-set records.
So, what to do? As you say this query is only run once a month I think it would only require a small change to business processes to have separate queries: one for all organisations and one for a sub-set of organisations.
"Btw, removing the :R1 IS NULL clause
doesn't change the execution plan
much, which leaves me with the other
side of the OR condition, :R1 <=
org.no where NULL wouldn't make sense
anyway, as org.no is NOT NULL"
Okay, so the thing is you have a pair of bind variables which specify a range. Depending on the distribution of values, different ranges might suit different execution plans. That is, this range would (probably) suit an indexed range scan...
WHERE org.id BETWEEN 10 AND 11
...whereas this is likely to be more fitted to a full table scan...
WHERE org.id BETWEEN 10 AND 1199999
That is where Bind Variable Peeking comes into play.
(depending on distribution of values, of course).
Since the query plans are actually consistently different, that implies that the optimizer's cardinality estimates are off for some reason. Can you confirm from the query plans that the optimizer expects the conditions to be insufficiently selective when bind variables are used? Since you're using 11.2, Oracle should be using adaptive cursor sharing so it shouldn't be a bind variable peeking issue (assuming you are calling the version with bind variables many times with different NO values in your testing.
Are the cardinality estimates on the good plan actually correct? I know you said that the statistics on the NO column are accurate but I would be suspicious of a stray histogram that may not be updated by your regular statistics gathering process, for example.
You could always use a hint in the query to force a particular index to be used (though using a stored outline or optimizer plan stability would be preferable from a long-term maintenance perspective). Any of those options would be preferable to resorting to dynamic SQL.
One additional test to try, however, would be to replace the SQL 99 join syntax with Oracle's old syntax, i.e.
SELECT <<something>>
FROM <<some other table>> cust,
organization org
WHERE cust.org_id = org.id
AND ( ((org_from IS NULL) OR (org_from <= org.no))
AND ((org_to IS NULL) OR (org_to >= org.no)))
That obviously shouldn't change anything, but there have been parser issues with the SQL 99 syntax so that's something to check.
It smells like Bind Peeking, but I am only on Oracle 10, so I can't claim the same issue exists in 11.
This looks a lot like a need for Adaptive Cursor Sharing, combined with SQLPlan stability.
I think what is happening is that the capture_sql_plan_baselines parameter is true. And the same for use_sql_plan_baselines. If this is true, the following is happening:
The first time that a query started it is parsed, it gets a new plan.
The second time, this plan is stored in the sql_plan_baselines as an accepted plan.
All following runs of this query use this plan, regardless of what the bind variables are.
If Adaptive Cursor Sharing is already active,the optimizer will generate a new/better plan, store it in the sql_plan_baselines but is not able to use it, until someone accepts this newer plan as an acceptable alternative plan. Check dba_sql_plan_baselines and see if your query has entries with accepted = 'NO' and verified = null
You can use dbms_spm.evolve to evolve the new plan and have it automatically accepted if the performance of the plan is at least 1,5 times better than without the new plan.
I hope this helps.
I added this as a comment, but will offer up here as well. Hope this isn't overly simplistic, and looking at the detailed responses I may be misunderstanding the exact problem, but anyway...
Seems your organisations table has column no (org.no) that is defined as a number. In your hardcoded example, you use numbers to do the compares.
JOIN organisations org
ON (cust.org_id = org.id
AND ((10 IS NULL) OR (10 <= org.no))
AND ((20 IS NULL) OR (20 >= org.no)))
In your procedure, you are passing in varchar2:
PROCEDURE DO_STUFF(
org_from VARCHAR2 := NULL,
org_to VARCHAR2 := NULL)
So to compare varchar2 to number, Oracle will have to do the conversions, so this may cause the full scans.
Solution: change proc to pass in numbers