I've written an mvc controller function to handle dynamic queries, it's working great but with some more complicated queries it's really bogging down as much as 15 second response time. I used ObjectQuery.ToTraceString to get the sql and execute it against my database and I'm getting 1-2 second response times, not great but hugely disparate from the time it takes for my server to respond to the same query.
Here is my code:
public class QueryController : Controller
{
//
// GET: /Query/
Entities1 Context = new Entities1();
public JsonResult Query(
string select,
string table = "Product",
string where = "",
string groupBy = ""
)
IQueryable dataTable;
if (table == "Customer") dataTable = Context.Customers;
else if (table == "Product") dataTable = Context.Products;
else if (table == "Purchase") dataTable = Context.Purchase;
else dataTable = Context.Farms;
if (select == null) return null;
string whereClause = where;
string selectClaus = select;
string groupByClaus = groupBy;
IQueryable queryResults = dataTable;
if (where != "") queryResults = queryResults.Where(whereClause);
if (groupByClaus != "") queryResults = queryResults.GroupBy(groupByClaus, "it");
queryResults = queryResults.Select(selectClaus);
JsonResult result = new JsonResult();
result.Data = queryResults;
result.JsonRequestBehavior = JsonRequestBehavior.AllowGet;
return result;
}
}
This code is supposed to handle requests like this:
?table=Purchase
&select=new (Customer.First as Name, Product.Id as ProdId)
&where=Category == 5
A query like the above takes about 700 ms, but if I try something more complicated it slows to a crawl (15 seconds):
?table=Purchase
&select=new (
count(true) as Total,
Key.Product as Product,
Key.CustomerRef as CustomerRef,
count(Price > 475) as BigPurchases,
count(PaidFor == false) as UnPaid,
count((Category != null) and (Comments == null) and Returns == null) as NoFeedback)
&groupBy=new (
Product.ProductName as Product,
CustomerRef as CustomerRef
)
In particular the navigation Property seems to be a problem, removing it speeds up the query considerably (3 sec):
?table=Purchase
&select=new (
count(true) as Total,
Key.Product as Product,
Key.CustomerRef as CustomerRef,
count(Price > 475) as BigPurchases,
count(PaidFor == false) as UnPaid,
count((Category != null) and (Comments == null) and Returns == null) as NoFeedback)
&groupBy=new (
ProductRef as Product,
CustomerRef as CustomerRef
)
The time all seems to get used iterating over the IEnumerable, in the code I provide I pass the data out and let the whatever underlying MVC code do whatever converting it wants, which takes approximately the time mentioned. However if I iterate over it myself, or use the ToList function I get similarly slow response times.
Does anyone have any ideas as to what is causing the long pause getting these entities?
Update
I added indexing to my database which sped things up, but it is still taking 20-30 times as long to execute the query in linq than it does in sql.
Related
I have a complex query that seems to cause a memory leak in my applications memory.
From my understanding the query result is cached so that this processing does not need to be done every time the query is executed. https://learn.microsoft.com/en-us/ef/core/querying/how-query-works
But what it looks like is that the query is getting cached for each and every user that logs into the system. From the memory dump it looks like we have thousands of compiled query cache objects for the same query.
The query looks as follows.
public async Task<IList<EmployeeInboxMessage>> GetEmployeeMessagesAsync(long employeeId)
{
return await (from message in this.Repository.Set
join userStep in this.Repository.Context.Set<UserWorkflowHeaderStep>() on message.UserWorkflowStepId equals userStep.UserWorkflowStepId into userSteps
from userStep in userSteps.DefaultIfEmpty()
join acceptedStep in this.Repository.Context.Set<CompanyWorkflowStep>() on userStep.AcceptedStepId equals acceptedStep.WorkflowStepId into acceptedSteps
from acceptedStep in acceptedSteps.DefaultIfEmpty()
join rejectedStep in this.Repository.Context.Set<CompanyWorkflowStep>() on userStep.RejectedStepId equals rejectedStep.WorkflowStepId into rejectedSteps
from rejectedStep in rejectedSteps.DefaultIfEmpty()
let step =
message.InboxEntryType == InboxEntryType.Claims ||
message.InboxEntryType == InboxEntryType.AdvancedLeave ||
message.InboxEntryType == InboxEntryType.ChangeRequest
? new WorkflowHeaderStep
{
WorkflowItem = userStep.WorkflowItem,
AcceptedStepId = userStep.AcceptedStepId,
AcceptedStep = acceptedStep == null ? null : new WorkflowStep
{
OnApprovalActionId = acceptedStep.OnApprovalActionId,
OnRejectionAction = acceptedStep.OnRejectionAction,
OrderNumber = acceptedStep.OrderNumber
},
RejectedStepId = userStep.RejectedStepId,
RejectedStep = rejectedStep == null ? null : new WorkflowStep
{
OnApprovalActionId = rejectedStep.OnApprovalActionId,
OnRejectionAction = rejectedStep.OnRejectionAction,
OrderNumber = rejectedStep.OrderNumber
}
}
: null
let employeeName = string.IsNullOrWhiteSpace(message.OBOEmployee.PreferredName)
? message.OBOEmployee.FullName
: message.OBOEmployee.PreferredName + " " + message.OBOEmployee.LastName
where message.EmployeeId == employeeId
orderby message.EffectiveDate
select new EmployeeInboxMessage
{
Message = message,
UserStep = step,
UserId = message.UserId,
UserName = message.User.FullName,
EmployeeName = message.OBOEmployeeId.HasValue
? employeeName
: message.User.FullName,
RelatedPrimaryKey =
message.InboxEntryType == InboxEntryType.Claims ||
message.InboxEntryType == InboxEntryType.AdvancedLeave ||
message.InboxEntryType == InboxEntryType.ChangeRequest
? userStep.RelatedPrimaryKey
: message.UserWorkflowStepId!.Value,
StartUserCompanyId = message.StartUser.CompanyId
}).ToListAsync();
}
It looks like your query is to complex for EF to create a single CompiledQueryCache object and it is adding a cached version for each "employeeId" that you are passing in. By the looks of it there is already 27804 versions.
If you restructure your query then it might solve your problem. Try removing the "let step = ..." and "let employeeName = ...". Worst case you would need to create raw SQL or maybe a view.
There is not a lot of documentation out there explaining how EF creates the query cache(I can't find any). You might never now what exactly your problem is except if some of the EF creators give some input or you could always go through their code.
If you still have a problem after changing your query it might be best to log a issue with efcore https://github.com/dotnet/efcore
My code is as below:
var conntionRecord1Id = (from connectionBase in orgServiceContext.CreateQuery("connection")
where connectionBase["record1roleid"] == null
select new { OpportunityId = connectionBase["record1id"] }).Distinct().ToList();
var query =
from opportunity in orgServiceContext.CreateQuery("opportunity")
orderby opportunity["createdon"] ascending
select new
{
Topic = opportunity.Attributes.Contains("name") == true ? opportunity["name"] : null,
OpportunityId = opportunity.Attributes.Contains("opportunityid") == true ? opportunity["opportunityid"] : null,
PostalCode = opportunity.Attributes.Contains("new_address_postalcode") == true ? opportunity["new_address_postalcode"] : null,
};
var result = (from f in query.ToList() where conntionRecord1Id.Contains(f.OpportunityId) select f).ToList();
But in above query where i am using Contains it giving count 0. even though I have common records in the list
The Contains Linq extension method does not know how to compare complex objects.
Comparing f.OpportunityId to a list of Guids instead of a list of an anonymous type will work:
var conntionRecord1Id = (from connectionBase in orgServiceContext
.CreateQuery("connection")
where connectionBase["record1roleid"] == null
select connectionBase.GetAttributeValue<Guid?>("record1id"))
.Distinct()
.ToList();
Implement IEqualityComparer in a custom comparer class to compare complex objects: https://stackoverflow.com/a/6694563/1817350
Keep in mind that calling .ToList() brings down all the records into memory and will cause performance problems with large amounts of records.
I have a SQL statement similar to:
SELECT DATEDIFF(Day, startDate, endDate) FROM Data WHERE ProjectId=#id
In the case where Data doesn't have any records for ProjectId, SQL Server returns null.
In Dapper, I execute this via:
value = conn.Query<int>("...").SingleOrDefault()
In this case, I expect the semantics of SingleOrDefault to mean "if this is null, return zero." In fact, my code is even more zero-friendly:
int toReturn = 0;
using (var conn = ...) {
toReturn = conn.Query<int>("SELECT DATEDIFF(...))");
}
return toReturn;
When I debug and step into this code, I find that the line yield return (T)func(reader) is throwing a null pointer exception.
Am I doing something wrong here, or is this by design?
(FYI, the work-around is to wrap my select in an ISNULL(..., 0))
In the case where Data doesn't have any records for ProjectId, SQL Server returns null.
In the case where Data doesn't have any matching records, SQL server does not really return null - it returns no rows. This scenario works fine:
var result = connection.Query<int>( // case with rows
"select DATEDIFF(day, GETUTCDATE(), #date)", new { date = DateTime.UtcNow.AddDays(20) })
.SingleOrDefault();
result.IsEqualTo(20);
result = connection.Query<int>( // case without rows
"select DATEDIFF(day, GETUTCDATE(), #date) where 1 = 0", new { date = DateTime.UtcNow.AddDays(20) })
.SingleOrDefault();
result.IsEqualTo(0); // zero rows; default of int over zero rows is zero
both of which work fine. The fact that you say ISNULL "fixes" it means that you are talking about a different scenario - the "I returned rows" scenario. Since that is the case, what your code is saying is "take this 1-or-more integers which contains a null, and map it as a non-nullable int" - that isn't possible, and the mapper is correct to throw an exception. Instead, what you want is:
int? result = connection.Query<int?>(...).SingleOrDefault();
Now, if there are rows, it is mapping the value to int?, and then applying the single-or-default. It you want that as an int, then maybe:
int result = connection.Query<int?>(...).SingleOrDefault() ?? 0;
If you need to be able to tell the difference between "zero rows" and "null result", then I would suggest:
class NameMe {
public int? Value {get;set;}
}
var row = connection.Query<NameMe>("select ... as [Value] ...", ...)
.SingleOrDefault();
if(row == null) {
// no rows
} else if(row.Value == null) {
// one row, null value
} else {
// one row, non-null value
}
or something similar
I have method where I need to retrieve using EF the last ten entries in the database and check to see if there is a match between the value and the current term. Here is what I have thus far
public static int ValidatePassword(string username, string password, int securityUserId)
{
int validResult = 0;
/*Need to pass to client a value based upon success or failure of validation
* 0 - success
* 1 - password has already been used in the last 10 entries
* 2 - password does not meet CJIS requirements
*/
IEnumerable<string> oldpassword = null;
// Create a Regular Expression to determine whether or not special characters are present.
Regex regularExpression = new Regex("[^a-z0-9]");
//if id exists pull last ten passwords
if (securityUserId > 0)
{
long id = Convert.ToInt64(securityUserId);
using (var context = new SecurityEntities(string.Empty))
{
try
{
oldpassword = (from p in context.SecurityAudits
where p.SecurityUserId == id &&
p.OldPassword == password
orderby p.ActionDate descending
select p.OldPassword.Take(10).ToString()).ToList();
}
catch (Exception ex)
{
string err = string.Format("ValidateCJISPassword() was unable to return an object msg:{0}", ex.Message);
throw new Exception(err, ex.InnerException);
}
finally
{
context.Dispose();
}
}
}
else if (oldpassword == null)
{
//no matching record found now check other requirements
if ((password.Length >= DEFAULT_CJIS_PASSWORD_MIN_LENGTH) && regularExpression.IsMatch(password) && (password != username))
{
//success
validResult = 0;
}
else
{
//password does not meet standard CJIS requirements
validResult = 2;
}
}
else
{
//matching record was found
validResult = 1;
}
return validResult;
}
}
Where I am currently hung up is the query throws an exception on the ToString() method
LINQ to Entities does not recognize the method 'System.String ToString()' method, and this method cannot be translated into a store expression.
I'm still learning EF and how linq works so I'm not sure what the best approach here is. Should I try to set the result to something other than IEnumerable like an array or List or is there another approach I should consider?
Thanks in advance,
Cheers,
Change this
oldpassword = (from p in context.SecurityAudits
where p.SecurityUserId == id &&
p.OldPassword == password
orderby p.ActionDate descending
select p.OldPassword.Take(10).ToString()).ToList();
To this
oldpassword = (from p in context.SecurityAudits
where p.SecurityUserId == id &&
p.OldPassword == password
orderby p.ActionDate descending
select p.OldPassword).Take(10).ToList();
The problem was that your Take(10) clause was not part of the whole result but inside the actual linq statement.. it goes on the outside of it to take the top 10 of the entire resultset.. then you do the ToList() which turns the whole thing into an array
The next problem is that you just created an array and assigned it to oldpassword
I don't see anything here that does anything with the array...
You need to do something like:
declare your array of strings
assign the array to the return of the linq query
evaluate the return for > 0 results
if > 0 then the password has been used in the last 10
if = 0 then new password should be ok, correct?
Now that I have an understanding of what I needed in the query I was able to also update the linq statement as follows:
var lastTenPassword = (from p in context.SecurityAudits.Take(10)
orderby p.ActionDate descending
where p.SecurityUserId == id
select p.OldPassword).ToList();
string oldpassword = lastTenPassword.Where(a => a == password).FirstOrDefault();
Testing is further down the line but now by moving the .Take() method inside the query I am explicitly grabbing the top ten where as my first attempt would have retrieved all the records and then grabbed the top ten.
For testing you can also see where I broke out the initial where() to first grab all records by id and then perform a filter on that set by looking for a matching password within that set.
Thanks again for your help
With this code:
i.SpesaAlloggio = db.TDP_NotaSpeseSezB.Sum(p => p.Costo / (((DateTime)p.DayEnd)
.Subtract((DateTime)p.DayStart).Days + 1));
I receive this error:
LINQ to Entities does not recognize the method
'System.TimeSpan Subtract(System.DateTime)' method, and this method cannot be
translated into a store expression.
How can I do this?
Use a calculated DB field and map that. Or use SqlFunctions with EF 4 as LukLed suggested (+1).
I wrote a function for removing time:
public static DateTime RemoveHours(DateTime date)
{
int year = date.Year;
int month = date.Month;
int day = date.Day;
return new DateTime(year, month, day);
}
and changed filtering condition:
var query =
from trn in context.IdentityTransactions
where trn.ClientUserId == userId && trn.DateDeleted == null
orderby trn.DateTimeCreated
select new
{
ClientServerTransactionID = trn.ClientServerTransactionID,
DateTimeCreated = trn.DateTimeCreated,
ServerTransDateTime = trn.ServerTransDateTime,
Timestamp = trn.Timestamp,
Remarc = trn.Remarc,
ReservedSum = trn.ReservedSum,
};
if (dateMin.HasValue && dateMin.Value > DateTime.MinValue)
{
DateTime startDate = Converters.RemoveHours(dateMin.Value);
query = from trn in query
where trn.DateTimeCreated >= startDate
select trn;
}
if (dateMax.HasValue && dateMax.Value > DateTime.MinValue)
{
var endDate = Converters.RemoveHours(dateMax.Value.AddDays(1.0));
query = from trn in query
where trn.DateTimeCreated < endDate
select trn;
}
dateMin and dateMax are nullable types and may be not set in my case.
Try (it is not very efficient, but it will work):
i.SpesaAlloggio = db.TDP_NotaSpeseSezB.ToList()
.Sum(p => p.Costo / (((DateTime)p.DayEnd)
.Subtract((DateTime)p.DayStart).Days + 1));
EDIT : This will be extremely slow for large tables, because it transfers whole table content form server
Entity Framework tries to translate your expression to SQL, but it can't handle ((DateTime)p.DayEnd).Subtract((DateTime)p.DayStart). You have to make it simpler. ToList() gets all rows and then makes the calculation on application side, not in database.
With EF4, you could use SqlFunctions DateDiff
With EF1, you could create calculated field or view with this field and make calculation based on this field.