I have the following object Job and Job sor are populated by reading in data from an XML file and Sor is populated from a database.
class Job
{
public int JobID { get; set; }
public string DepartmentCode { get; set; }
public string ClientReference { get; set; }
public string JobDescription { get; set; }
public List<JobSor> JobSorList { get; set; }
}
class JobSor
{
public int JobID { get; set; }
public string SorUserCode { get; set; }
public string SorNotes1 { get; set; }
public string SorNotes2 { get; set; }
}
class Sor
{
[Key]
public string code { get; set; }
public string description { get; set; }
public string contract { get; set; }
}
I want to write a linq query that will show me all the JobSors that do not exist in the Sor object.
This is what I have so far but I can’t reference the SorUserCode property?
var db = new dbContext();
var sor = db.Sors.Where(p => p.contract == "??");
var query =
from j in jobs
join p in sor on j.JobSorList.SorUserCode equals p.code into jp
from a in jp.DefaultIfEmpty()
select j;
How can I do this?
First get list of all JobSor from jobs list.
Then apply condition Where its SorUserCode value does not match with Any code value of the sor list.
Your query will be as below.
var query = jobs.SelectMany(x => x.JobSorList)
.Where(x => !sor.Any(y => y.code == x.SorUserCode));
I have a SimID query that joins a given list of cell phone numbers with their respective IDs in my DB, if the number doesn't exist in the DB it returns 0 as an ID. If I watch my query while debugging, it executes perfectly and very quickly. However when I loop through the results of the query and create a new object and add it to my model it takes 3 minutes to create 458 new objects.
I am new to EF and Linq what am I doing wrong? How can I optimize this code to execute faster?
Any help would be greatly appreciated.
//Query to create virtual table of SimID and MSISDN joined on Cache.MSISDN
var SimID = (from ca_msisdn in Cache.MSISDN
join db_simobj in ctx.Sims on ca_msisdn equals db_simobj.Msisdn into Holder
from msisdnresult in Holder.DefaultIfEmpty()
select new { MSISDN = ca_msisdn, ID = (msisdnresult == null || msisdnresult.SimId == 0 ? 0 : msisdnresult.SimId) });
//Loop through virtual tables and add new data to model
foreach (var ToUpdate in SimID)
{
if (ToUpdate.ID == 0)
{
Console.WriteLine("We have found a new MSISDN: " + ToUpdate.MSISDN + " adding it to the model.");
ctx.Sims.Add(new Sim { Msisdn = ToUpdate.MSISDN });
}
}
//My Sim object
public partial class Sim
{
public Sim()
{
this.CDR_Event = new HashSet<CDR_Event>();
}
public long SimId { get; set; }
public long SimStatusId { get; set; }
public Nullable<long> FitmentCentreId { get; set; }
public string Serial { get; set; }
public string Msisdn { get; set; }
public string Puk { get; set; }
public string Network { get; set; }
public Nullable<System.DateTime> SimStatusDate { get; set; }
public Nullable<System.DateTime> ActivationDate { get; set; }
public Nullable<System.DateTime> ExpiryDate { get; set; }
public Nullable<long> APNSimStatusId { get; set; }
public Nullable<System.DateTime> APNActivated { get; set; }
public Nullable<System.DateTime> APNConfirmed { get; set; }
public string Svr { get; set; }
public virtual ICollection<CDR_Event> CDR_Event { get; set; }
}
Your main problem is that every time that you add to the Sims collection on your context you are adding to what the context's change tracker needs to keep track of, which can become a very expensive proposition memory-wise when you start having larger amounts of entities.
You can solve this in one of two ways:
1) Save the data in batches. That way the change tracker never has to keep track of a large amount of added entities.
2) If you don't want to save it right away, keep it as an in-memory list first, and then do #1 when you do go to save those records.
The answer to this question as IronMan84 alluded to; as documented here is to use .AddRange() and add everything all at once as a list outside the foreach instead of one by one inside the loop.
List<Sim> _SIMS = new List<Sim>();
foreach (var ToUpdate in SimID)
{
if (ToUpdate.ID == 0)
{
Console.WriteLine("We have found a new MSISDN: " + ToUpdate.MSISDN + " adding it to the model.");
_SIMS.Add(new Sim { Msisdn = ToUpdate.MSISDN} );
}
}
ctx.Sims.AddRange(_SIMS);
I am trying to find one or more documents in RavenDB based on the values of a child collection.
I have the following classes
public class GoldenDocument
{
public GoldenDocument()
{
LinkedDocuments = new List<LinkedDocument>();
MergeMatchFields = new List<MergeMatchField>();
}
public string Id { get; set; }
public Guid SourceRowId { get; set; }
public List<MergeMatchField> MergeMatchFields { get; set; }
public List<LinkedDocument> LinkedDocuments { get; set; }
}
And the class that is in the collection MergeMatchFields
public class MergeMatchField
{
public string Id { get; set; }
public Guid OriginId { get; set; }
public string Name { get; set; }
public MatchType MatchType { get; set; }
public double MatchPerc { get; set; }
public string Value { get; set; }
}
In a List<MergeFields> mergeFields collection I have values that is not stored in RavenDB yet. Values are compared to values in a RavenDB document for find if it is a possible match by executing the following query:
using (var session = documentStore.OpenSession())
{
var docs = from gd in session.Query<GoldenDocument>()
from mf in gd.MergeMatchFields
from tf in mergeFields
where mf.Name == tf.Name
&& JaroWinklerCalculator.jaroWinkler(mf.Value, tf.Value) > .90d
&& !string.IsNullOrEmpty(mf.Value)
select gd;
}
I understand that ravenDB does not support SelectMany() so how would I go about getting the results from the Document store?
Create an index for this that would output the values you want to query on.
Note that you can't just execute arbitrary code the way you do here: JaroWinklerCalculator.jaroWinkler(mf.Value, tf.Value) > .90d
But you can use fuzzy queries, and they will do the same.
Below is the code I am using and the database table it is pulling from has about 92000 records in it. The way it is pulling right now it is pulling all 92000 records then doing the filtering.
What I am looking to do is the filtering on the initial pull from the DB so that it does not take aproximately 40 seconds to load the page.
This is something I am still new at so I am lost as to how to do this and make it work with my view
public ViewResult Makes()
{
var items = (from item in DBCacheHelper.recallslist
orderby item.MFGTXT ascending
select item.ToDomainRecall()).GroupBy(item => item.MFGTXT).Select(grp => grp.First());
return View(items);
}
public static IEnumerable<Recall> recallslist
{
get
{
if (c["GetAllRecalls"] == null)
{
c.Insert("GetAllRecalls", GetAllRecalls());
return (IEnumerable<Recall>)c["GetAllRecalls"];
}
else
{
return (IEnumerable<Recall>)c["GetAllRecalls"];
}
}
}
public static IEnumerable<Recall> GetAllRecalls()
{
using (DealerContext context = new DealerContext())
{
var items = from item in context.recalls.ToList<Recall>()
select item.ToDomainRecall();
return items.ToList<Recall>();
}
}
SELECT
[Extent1].[RecallsId] AS [RecallsId],
[Extent1].[RECORD_ID] AS [RECORD_ID],
[Extent1].[CAMPNO] AS [CAMPNO],
[Extent1].[MAKETXT] AS [MAKETXT],
[Extent1].[MODELTXT] AS [MODELTXT],
[Extent1].[YEARTXT] AS [YEARTXT],
[Extent1].[MFGCAMPNO] AS [MFGCAMPNO],
[Extent1].[COMPNAME] AS [COMPNAME],
[Extent1].[MFGNAME] AS [MFGNAME],
[Extent1].[BGMAN] AS [BGMAN],
[Extent1].[ENDMAN] AS [ENDMAN],
[Extent1].[RCLTYPECD] AS [RCLTYPECD],
[Extent1].[POTAFF] AS [POTAFF],
[Extent1].[ODATE] AS [ODATE],
[Extent1].[INFLUENCED_BY] AS [INFLUENCED_BY],
[Extent1].[MFGTXT] AS [MFGTXT],
[Extent1].[RCDATE] AS [RCDATE],
[Extent1].[DATEA] AS [DATEA],
[Extent1].[RPNO] AS [RPNO],
[Extent1].[FMVSS] AS [FMVSS],
[Extent1].[DESC_DEFECT] AS [DESC_DEFECT],
[Extent1].[CONEQUENCE_DEFECT] AS [CONEQUENCE_DEFECT],
[Extent1].[CORRECTIVE_ACTION] AS [CORRECTIVE_ACTION],
[Extent1].[NOTES] AS [NOTES],
[Extent1].[RCL_CMPT_ID] AS [RCL_CMPT_ID]
FROM [dbo].[Recalls] AS [Extent1]
Update:
Ultimately I would like to only pull records from the Recalls Table where the MFGTXT is equal to the
MakeName in the AutoMake Table
public class AutoMake
{
[Key]
public int MakeID { get; set; }
public string MakeName { get; set; }
public AutoMake ToDomainAutoMakes()
{
return new AutoMake
{
MakeID = this.MakeID,
MakeName = this.MakeName
};
}
}
public class Recall
{
[Key]
public int RecallsId { get; set; }
public string RECORD_ID { get; set; }
public string CAMPNO { get; set; }
public string MAKETXT { get; set; }
public string MODELTXT { get; set; }
public string YEARTXT { get; set; }
public string MFGCAMPNO { get; set; }
public string COMPNAME { get; set; }
public string MFGNAME { get; set; }
public string BGMAN { get; set; }
public string ENDMAN { get; set; }
public string RCLTYPECD { get; set; }
public string POTAFF { get; set; }
public string ODATE { get; set; }
public string INFLUENCED_BY { get; set; }
public string MFGTXT { get; set; }
public string RCDATE { get; set; }
public string DATEA { get; set; }
public string RPNO { get; set; }
public string FMVSS { get; set; }
public string DESC_DEFECT { get; set; }
public string CONEQUENCE_DEFECT { get; set; }
public string CORRECTIVE_ACTION { get; set; }
public string NOTES { get; set; }
public string RCL_CMPT_ID { get; set; }
public Recall ToDomainRecall()
{
return new Recall
{
RECORD_ID = this.RECORD_ID,
CAMPNO = this.CAMPNO,
MAKETXT = this.MAKETXT,
MODELTXT = this.MODELTXT,
YEARTXT = this.YEARTXT,
MFGCAMPNO = this.MFGCAMPNO,
COMPNAME = this.COMPNAME,
MFGNAME = this.MFGNAME,
BGMAN = this.BGMAN,
ENDMAN = this.ENDMAN,
RCLTYPECD = this.RCLTYPECD,
POTAFF = this.POTAFF,
ODATE = this.ODATE,
INFLUENCED_BY = this.INFLUENCED_BY,
MFGTXT = this.MFGTXT,
RCDATE = this.RCDATE,
DATEA = this.DATEA,
RPNO = this.RPNO,
FMVSS = this.FMVSS,
DESC_DEFECT = this.DESC_DEFECT,
CONEQUENCE_DEFECT = this.CONEQUENCE_DEFECT,
CORRECTIVE_ACTION = this.CORRECTIVE_ACTION,
NOTES = this.NOTES,
RCL_CMPT_ID = this.RCL_CMPT_ID
};
}
}
If you want to add server side filtering outside of your repository methods, you need to return your types as IQueryable rather than IEnumerable and not call .ToList, .AsEnumerable, or any other method that would cause .GetEnumerator to be called. Additionally, your cast `(IEnumerable)c["GetAllRecalls"];' forces LINQ to Objects to be used for subsequent requests rather than retaining the expression tree and using Entity Framework. That being said, you may need to move your call to ToDomainRecall method to after the additional filter is applied as well because that can't be translated to your database. Here are some of the changes you would need to make:
public ViewResult Makes()
{
var items = (from item in DBCacheHelper.recallslist
orderby item.MFGTXT ascending
select item.ToDomainRecall()).GroupBy(item => item.MFGTXT).Select(grp => grp.First());
return View(items);
}
public static IQueryable<Recall> recallslist
{
get
{
if (c["GetAllRecalls"] == null)
{
c.Insert("GetAllRecalls", GetAllRecalls(context));
}
return c["GetAllRecalls"];
}
}
public static IQueryable<Recall> GetAllRecalls(DealerContext context)
{
var items = context.recalls;
return items;
}
Looks like your DBaccess is done in the call to DBCacheHelper.recallslist.
You need to edit the sql that runs from/in this function.
As Eranga pointed out, you don't show how you are filtering the large number down to a smaller number of records. I assume you want 20 or 100 at a time? If so, please see the accepted answer here:
efficient way to implement paging
Specifically, this part which shows how to only retrieve rows x to y (where x = #p0 + 1 AND y = #p0 + #p1):
SELECT [t1].[CodCity],
[t1].[CodCountry],
[t1].[CodRegion],
[t1].[Name],
[t1].[Code]
FROM (
SELECT ROW_NUMBER() OVER (
ORDER BY [t0].[CodCity],
[t0].[CodCountry],
[t0].[CodRegion],
[t0].[Name],
[t0].[Code]) AS [ROW_NUMBER],
[t0].[CodCity],
[t0].[CodCountry],
[t0].[CodRegion],
[t0].[Name],
[t0].[Code]
FROM [dbo].[MtCity] AS [t0]
) AS [t1]
WHERE [t1].[ROW_NUMBER] BETWEEN #p0 + 1 AND #p0 + #p1
ORDER BY [t1].[ROW_NUMBER]
I have this POCO and I want to return a list of the users in a particular company.
public class Company
{
public AccreditedCompany()
{
this.Branches = new HashSet<Branch>();
}
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity), ScaffoldColumn(false)]
public int CompanyId { get; set; }
public bool Active { get; set; }
public virtual ICollection<Branch> Branches { get; set; }
}
public class Branch
{
public Branch()
{
this.Users = new HashSet<User>();
}
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity), ScaffoldColumn(false)]
public int BranchId { get; set; }
public int CompanyId { get; set; }
public string Name { get; set; }
public string ContactName { get; set; }
public virtual Company Company { get; set;}
public virtual ICollection<User> Users { get; set; }
}
public class User
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity), ScaffoldColumn(false)]
public int UserId { get; set; }
public int BranchId { get; set; }
public string ComputerSN { get; set; }
public string CameraSN { get; set; }
public virtual Branch Branch { get; set; }
}
This is my LINQ query:
var company = (from u in objDataContext.Companies.Include(c=>c.Branches.Select(v=>v.Users))
where u.CompanyId == 8 select u).FirstOrDefault();
IQueryable<User> users = (from j in company.Branches select j.Users);
I have this compilation error on the second query:
Error 2 Cannot implicitly convert type
'System.Collections.Generic.IEnumerable>'
to 'System.Linq.IQueryable'. An explicit conversion exists (are
you missing a cast?)
I want to get a list of the users, similar to a plain SQL statement like
SELECT dbo.Users.* FROM Branches
INNER JOIN dbo.Users ON dbo.Branches.BranchId = dbo.Users.BranchId
INNER JOIN dbo.Companies ON dbo.Branches.CompanyId = dbo.Companies.CompanyId
WHERE (dbo.Companies.CompanyId = 8)
Thanks in advance.
Your user query could be:
IEnumerable<User> users = company.Branches.SelectMany(branch => branch.Users);
This will return all users in any branch of the company.
It looks to me like you could just use:
IQueryable<User> users = objDataContext.Users
.Where(u => u.Branch.CompanyId == 8);
I notice you have both Company and CompanyId on your Branch entity, though. That seems redundant, even though it simplifies this query slightly. You should be able to get rid of Branch.CompanyId and User.BranchId and just use the entity associations.