I'm trying to rewrite a direct (disconnected) DataSet.DataTable.Select to LINQ for a textual search:
string search = "ProductNo like '%" + searchpattern +
"%' OR ProductName like '%" + searchpattern +
"%' OR Description like '%" + searchpattern + "%'";
DataSetProducts.sk_productsRow[] dsp = (DataSetProducts.sk_productsRow[])dsProducts.sk_products.Select(search, sort);
This works really fast. However if I re-enact it in LINQ:
productlist = from prds in dsProducts.sk_products.AsEnumerable()
where (prds.Field<string>("productno").Contains(searchpattern) ||
prds.Field<string>("productname").Contains(searchpattern) ||
prds.Field<string>("description").Contains(searchpattern))
select prds;
It becomes really slow because of the text-operations, and will return far less rows.
How can I achieve same results with the same speed? I have only ~1000 records in the table.
I've never used it myself, but perhaps i4o - Indexed LINQ might help speed things up...
As far as why it returns fewer rows, is it a case-sensitivity issue? (i.e. "like" is not case-sensitive by default; "Contains" is, by default)
If you are running within a LINQ to SQL database context, use the System.Data.Linq.SqlClient.SqlMethods.Like operation instead of String.Contains.
The reason you are seeing the difference in speed when you execute the LINQ query versus the disconnected dataset is because the dataset is already in memory while the LINQ query is executed at time of use.
Other than that if you haven't installed the LINQ Visualizer which will show you the query that is actually being executed.
Dumb question, but is the .AsEnumerable() necessary for something? I'm only experienced so far with Linq2Sql, but I would normally treat things as AsQueryable as long as possible in order to avoid the "load dataset, then immediately filter it down again" scenario.
Related
Hey i have the following query which i am executing via an autowired jdbctemplate in my spring-boot application:
query = "select fromfilepath from filecopyeventlog where copytype = 1 ORDER BY copytimestamp DESC LIMIT 1"
i execute the query via:
String result = jdbcTemplate.queryForObject(query, String.class);
since the result is just a single column defined as varchar(250).
Now the problem is, that this query is very slow when executed in my application - it takes many minutes..
when i execute the query direcly on the database webinterface (h2), i get the result in less then 900ms.
does someone have any idea? The result looks like:
/data/internal/s3/sensor-data/semiconductor_2019-07-12T01_17_11Z.mf4.gz
in the database are more then 500.000 rows.
I think the query is fine it seems like something in the result mapping? I thought maybe the backslashes are bad and changed them, to "_" but that resulted in no change.
hope someone can help :)
We have a database with some fields that are varchar(max) which could contain lots of text however I have a situation where I only want to select the first for example 300 characters from the field for a paginated table of results on a MVC web site for a "preview" of the field.
for a simplified example query where I want to get all locations to display in the table
(this would be paginated, so I don't just get everything - I get maybe 10 results at a time):
return db.locations;
However this gives me a location object with all the fields containing the massive amounts of text which is very time consuming to execute.
So what I resorted to before was using SQL stored procedures with the:
LEFT(field, 300)
to resolve this issue and then in the Linq to SQL .dbml file included the stored procedure to return a "location" object for the result.
However I have many queries and I don't want to have to do this for every query.
This maybe a simple solution, but I am not sure how I can phrase this on a search engine, I would appreciate anyone who can help me with this problem.
You can use functions that directly translate to those functions too, this is useful when you need to translate code that functionally works just fine in SQL at no risk in LINQ.
Have a look at System.Data.Objects.EntityFunctions
Locations.Select(loc=>System.Data.Objects.EntityFunctions.Left(loc.Field,300))
This will get directly translated into a LEFT on the server side.
EDIT: I misread LEFT for LTRIM. Here's all the String functions that can't be used in LINQ to SQL. Have you tried String.Substring()?
Your best option is to map the stored procedure and continue using it. Here is an excellent article with screen shots showing you how to do so.
If you're not using the designer tool you can also call ExecuteCommand against the DataContext. It isn't pretty, but it's what we have for now.
I found something like this worked for me:
return from locationPart in db.locations
select new LocationPart
{
Description = locationPart.description,
Text = locationPart.text.Substring(0,300)
};
Not ideal because I have to use "select new" to return a a different object, but it seems to work.
suppose my LINQ query is like
var qry = from c in nwEntitiesContext.CategorySet.AsEnumerable()
let products = this.GetProducts().WithCategoryID(c.CategoryID)
select new Model.Category
{
ID = c.CategoryID,
Name = c.CategoryName,
Products = new Model.LazyList<Core.Model.Product>(products)
};
return qry.AsQueryable();
i just want to know what query it will generate at runtime....how to see what query it is generating from VS2010 IDE when we run the code in debug mode....guide me step by step.
There is not much to see here - it will just select all fields from the Category table since you call AsEnumerable thus fetching all the data from the Category table into memory. After that you are in object space. Well, depending on what this.GetProducts() does - and my guess it makes another EF query fetching the results into memory. If that's the case, I would strongly recommend you to post another question with this code and the code of your GetProducts method so that we can take a look and rewrite this in a more optimal way. (Apart from this, you are projecting onto a mapped entity Model.Category which again won't (and should not) work with Linq-to-Entities.)
Before reading into your query I was going to recommend doing something like this:
string sqlQueryString = ((ObjectQuery)qry).ToTraceString();
But that won't work since you are mixing Linq-to-Entities with Linq-to-objects and you will actually have several queries executed in case GetProducts queries EF. You can separate the part with your EF query and see the SQL like this though:
string sqlString = nwEntitiesContext.CategorySet.ToTraceString();
but as I mentioned earlier - that would just select everything from the Categories table.
In your case (unless you rewrite your code in a drastic way), you actually want to see what queries are run against the DB when you execute the code and enumerate the results of the queries. See this question:
exact sql query executed by Entity Framework
Your choices are SQL Server Profiler and Entity Framework Profiler. You can also try out LinqPad, but in general I still recommend you to describe what your queries are doing in more detail (and most probably rewrite them in a more optimal way before proceeding).
Try Linqpad
This will produce SELECT * FROM Categories. Nothing more. Once you call AsEnumerable you are in Linq-to-objects and there is no way to get back to Linq-to-entities (AsQueryable doesn't do that).
If you want to see what query is generated use SQL Profiler or any method described in this article.
I'm tasked with adding an option to our search, which will return results where a given field doesn't begin with a letter of the alphabet. (The .StartsWith(letter) part wasn't so hard).
But I'm rather unsure about how to get the results that don't fall within the A-Z set, and equally hoping it generates some moderately efficient SQL underneath.
Any help appreciated - thanks.
In C# use the following construct, assuming db as a data context:
var query = from row in db.SomeTable
where !System.Data.Linq.SqlClient.SqlMethods.Like(row.SomeField, "[A-Z]%")
select row;
This is only supported in LINQ to SQL queries. All rules of the T-SQL LIKE operator apply.
You could also use less effective solution:
var query = from row in db.SomeTable
where row.SomeField[0] < 'A' || row.SomeField[0] > 'Z'
select row;
This gets translated into SUBSTRING, CAST, and UNICODE constructs.
Finally, you could use VB, where there appears to be a native support for the Like method.
Though SQL provides the ability to check a range of characters in a LIKE statement using bracket notation ([a-f]% for example), I haven't seen a linq to sql construct that supports this directly.
A couple thoughts:
First, if the result set is relatively small, you could do a .ToList() and filter in memory after the fact.
Alternatively, if you have the ability to change the data model, you could set up additional fields or tables to help index the data and improve the search.
--EDIT--
Made changes per Ruslan's comment below.
Well, I have no idea if this will work because I have never tried it and don't have a compiler nearby to try it, but the first thing I would try is
var query = from x in db.SomeTable
where x.SomeField != null &&
x.SomeField.Length >= 1 &&
x.SomeField.Substring(0, 1).All(c => !Char.IsLetter(c))
select x;
The possiblility exists that LINQ to SQL fails to convert this to SQL.
I started playing around with Linq today and ran into a problem I couldn't find an answer to. I was querying a simple SQL Server database that had some employee records. One of the fields is the full name (cn). I thought it would be interesting to group by the first name by splitting the full name at the first space. I tried
group by person.cn.Split(separators)[0]
but ran into a lengthy runtime exception (looked a lot like a C++ template instantiation error).
Then I tried grouping by a few letters of the first name:
group by person.cn.Substring(0,5)
and that worked fine but is not what I want.
I'm wondering about two things:
Why does the first example not work when it looks so close to the second?
Knowing that behind the scenes it's SQL stuff going on, what's a good way to do this kind of thing efficiently
Thanks,
Andrew
Split has no translation into SQL.
So, how to do this string manipulation without split? Cheat like hell (untested):
string oneSpace = " ";
string fiftySpace = " ";
var query =
from person in db.Persons
let lastname = person.cn.Replace(oneSpace, fiftySpace).SubString(0, 50).Trim()
group person by lastname into g
select new { Key = g.Key, Count = g.Count };
The reason your first attempt didn't work is because LINQ to SQL uses Expression Trees to translate your query into SQL. As a result any code that isn't directly translatable into SQL is an exception - this includes the call to Split.
Thanks guys, I'll try the "Replace" trick to see if that runs. I'm very intrigued by LINQ but now it looks like there's some hidden mysteriousness where you have to know what your LINQ queries translate into before being able to use it effectively.
The core problem is of course that I don't know SQL very well so that's where I'll start.
Edit:
I finally tried the "Replace" today and it works. I even got to sort the grouped results by count so now I have a pareto of name in my company. It's horrendously slow, though. Much faster to select everything and do the bucketing in C# directly.
Thanks again,
Andrew