Why is adding objects to my model so slow? - performance

I have a SimID query that joins a given list of cell phone numbers with their respective IDs in my DB, if the number doesn't exist in the DB it returns 0 as an ID. If I watch my query while debugging, it executes perfectly and very quickly. However when I loop through the results of the query and create a new object and add it to my model it takes 3 minutes to create 458 new objects.
I am new to EF and Linq what am I doing wrong? How can I optimize this code to execute faster?
Any help would be greatly appreciated.
//Query to create virtual table of SimID and MSISDN joined on Cache.MSISDN
var SimID = (from ca_msisdn in Cache.MSISDN
join db_simobj in ctx.Sims on ca_msisdn equals db_simobj.Msisdn into Holder
from msisdnresult in Holder.DefaultIfEmpty()
select new { MSISDN = ca_msisdn, ID = (msisdnresult == null || msisdnresult.SimId == 0 ? 0 : msisdnresult.SimId) });
//Loop through virtual tables and add new data to model
foreach (var ToUpdate in SimID)
{
if (ToUpdate.ID == 0)
{
Console.WriteLine("We have found a new MSISDN: " + ToUpdate.MSISDN + " adding it to the model.");
ctx.Sims.Add(new Sim { Msisdn = ToUpdate.MSISDN });
}
}
//My Sim object
public partial class Sim
{
public Sim()
{
this.CDR_Event = new HashSet<CDR_Event>();
}
public long SimId { get; set; }
public long SimStatusId { get; set; }
public Nullable<long> FitmentCentreId { get; set; }
public string Serial { get; set; }
public string Msisdn { get; set; }
public string Puk { get; set; }
public string Network { get; set; }
public Nullable<System.DateTime> SimStatusDate { get; set; }
public Nullable<System.DateTime> ActivationDate { get; set; }
public Nullable<System.DateTime> ExpiryDate { get; set; }
public Nullable<long> APNSimStatusId { get; set; }
public Nullable<System.DateTime> APNActivated { get; set; }
public Nullable<System.DateTime> APNConfirmed { get; set; }
public string Svr { get; set; }
public virtual ICollection<CDR_Event> CDR_Event { get; set; }
}

Your main problem is that every time that you add to the Sims collection on your context you are adding to what the context's change tracker needs to keep track of, which can become a very expensive proposition memory-wise when you start having larger amounts of entities.
You can solve this in one of two ways:
1) Save the data in batches. That way the change tracker never has to keep track of a large amount of added entities.
2) If you don't want to save it right away, keep it as an in-memory list first, and then do #1 when you do go to save those records.

The answer to this question as IronMan84 alluded to; as documented here is to use .AddRange() and add everything all at once as a list outside the foreach instead of one by one inside the loop.
List<Sim> _SIMS = new List<Sim>();
foreach (var ToUpdate in SimID)
{
if (ToUpdate.ID == 0)
{
Console.WriteLine("We have found a new MSISDN: " + ToUpdate.MSISDN + " adding it to the model.");
_SIMS.Add(new Sim { Msisdn = ToUpdate.MSISDN} );
}
}
ctx.Sims.AddRange(_SIMS);

Related

Unable to get Item from SQLite.Net Async PCL

I am struggling to get an Item by ID using the asynchronous API of SQLite.Net Async PCL. Here is my model class
public class Invoice : IEntityBase
{
public Invoice()
{
LineItems = new List<LineItem>();
DateCreated = DateTime.Now;
}
[PrimaryKey, AutoIncrement, Column("_id")]
public int Id { get; set; }
public DateTime DateCreated { get; set; }
public int Term { get; set; }
public bool Paid { get; set; }
public decimal Total { get; set; }
public string Notes { get; set; }
[OneToMany(CascadeOperations = CascadeOperation.All)]
public List<LineItem> LineItems { get; set; }
}
And the LineItems that has a One to Many relationship here
[PrimaryKey, AutoIncrement, Column("_id")]
public int Id { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public decimal Price { get; set; }
public string Category { get; set; }
public int Qty { get; set; }
[ForeignKey(typeof(Invoice))]
public int InvoiceId { get; set; }
[ManyToOne]
public Invoice Invoice { get; set; }
Here is the constructor:
public SQLiteAsyncConnection DbConnection;
public InvoiceDatabase(ISQLitePlatform platform, string databasePath)
{
if (DbConnection == null)
{
var connectionAsync = new Func<SQLiteConnectionWithLock>(() =>
new SQLiteConnectionWithLock
(
platform,
new SQLiteConnectionString(databasePath, false)
)
);
DbConnection = new SQLiteAsyncConnection(connectionAsync);
DbConnection.CreateTableAsync<Invoice>();
DbConnection.CreateTableAsync<LineItem>();
}
}
Other CRUD methods (Insert, GetALL) is working except getting an Invoice by ID, and both Visual Studio and Xamarin Studio are not giving me any useful stacktrace.
Here is the Get Method
private readonly InvoiceDatabase _database;
public InvoiceRepository(ISQLitePlatform platform, string databasePath)
{
if (_database == null)
{
_database = new InvoiceDatabase(platform, databasePath);
}
}
public async Task<Invoice> GetInvoice(int id)
{
var result = await _database.DbConnection.Table<Invoice>()
.Where(t => t.Id == id)
.FirstOrDefaultAsync();
return result;
}
I am passing in the Android implementation of SQLite, and like I said the Database is created but I am unable to get the Invoice object back, I even tried
public Task<Invoice> GetInvoiceWithChildren(int id)
{
return _database.DbConnection.GetWithChildrenAsync<Invoice>(id);
}
Any Help will be greatly appreciated.
After three days of chasing shadows it turned out that it is just a very simple thing that is tripping me up. I am tying to save a List of objects like so
[OneToMany(CascadeOperations = CascadeOperation.All)]
public List<LineItem> LineItems { get; set; }
I missed the part of the documentation that repeats the fact that SQLite.Net is a lightweight ORM - that point could not be stressed enough so you will have to remove your full size ORM hats such EF. So after reading from the SQLite-Net Extension documentation which says
Text blobbed properties
Text-blobbed properties are serialized into a text property when saved and deserialized when loaded. This allows storing simple objects in the same table in a single column.
Text-blobbed properties have a small overhead of serializing and deserializing the objects and some limitations, but are the best way to store simple objects like List or Dictionary of basic types or simple relationships.
I change my proptery like so and everything is now working as expected. Off now to dealing with the nuances of Async and Await
[TextBlob("LineItemBlobbed")]
public List<LineItem> LineItems { get; set; }
public string LineItemBlobbed { get; set; }

Entity Framework: Custom proxies for POCO

I am working with the EF6 and I am a big fan of the dynamic proxies, which enables lazy loading and change tracking. Anyway I am not happy, that the lazy loading is triggered once the property is accessed instead of loading the data, when the enumerator or the count property is called first. Therefore I tried to diesable the proxys and replace them by custom proxies. It was an easy thing to use a custom object context and overload the CreateObject method. Unfortantly the ObjectMaterialized event cannot replace the entity and I am not able to replace an entity from a query. The creation of the object lies deep in internal classes of the framework.
Has anybody an idea how to use custom proxies? Or how I am able to replace the entities materialized in an object query?
You should .Include the properties you want to fetch so that you avoid an N+1 query problem.
public class User
{
public int Id { get; set; }
public string Name { get; set ;}
public virtual ICollection<Post> Posts { get; set; }
public virtual ICollection<Comment> Comments { get; set; }
}
public class Post
{
public int Id { get; set; }
public string Title { get; set ; }
public int AuthorId { get; set; }
public virtual User Author { get; set; }
public virtual ICollection<Comment> Comments { get; set; }
}
public class Comment
{
public int Id { get; set; }
public string Note { get; set ;}
public int PostId { get; set; }
public virtual Post Post { get; set; }
public int AuthorId { get; set; }
public virtual User Author { get; set; }
}
public class BlogContext : DbContext
{
public DbSet<User> Users { get; set; }
public DbSet<Post> Posts { get; set; }
public DbSet<Comment> Comments { get; set; }
}
Then this is BAD in that it'll do tons of queries:
using (var db = new BlogContext())
{
var user = db.Users.Single(u => u.Id=5)); // +1 query
foreach (var post in user.Posts) // N queries
{
var message = String.Format("{0} wrote {1}", user.Name, post.Title);
Console.WriteLine(message);
foreach (var comment in post.Comments) // N * M queries!
{
// and that .Author make N * M MORE!
var message = String.Format("\t{0} commented {1}", comment.Author.Name, comment.Note);
Console.WriteLine(message);
}
}
}
And this is GOOD in that it'll do one query:
using (var db = new BlogContext())
{
var user = db.Users
.Single(u => u.Id=5))
.Include(u => u.Posts) // eliminates the N post queries
.Include(u => u.Posts.Comments) // eliminates the M comment queries
.Include(u => u.Posts.Comments.Author); // eliminates the M comment author queries
foreach (var post in user.Posts) // N queries
{
var message = String.Format("{0} wrote {1}", user.Name, post.Title);
Console.WriteLine(message);
foreach (var comment in post.Comments) // N * M queries!
{
// and that .Author make N * M MORE!
var message = String.Format("\t{0} commented {1}", comment.Author.Name, comment.Note);
Console.WriteLine(message);
}
}
}

RavenDB SelectMany not supported

I am trying to find one or more documents in RavenDB based on the values of a child collection.
I have the following classes
public class GoldenDocument
{
public GoldenDocument()
{
LinkedDocuments = new List<LinkedDocument>();
MergeMatchFields = new List<MergeMatchField>();
}
public string Id { get; set; }
public Guid SourceRowId { get; set; }
public List<MergeMatchField> MergeMatchFields { get; set; }
public List<LinkedDocument> LinkedDocuments { get; set; }
}
And the class that is in the collection MergeMatchFields
public class MergeMatchField
{
public string Id { get; set; }
public Guid OriginId { get; set; }
public string Name { get; set; }
public MatchType MatchType { get; set; }
public double MatchPerc { get; set; }
public string Value { get; set; }
}
In a List<MergeFields> mergeFields collection I have values that is not stored in RavenDB yet. Values are compared to values in a RavenDB document for find if it is a possible match by executing the following query:
using (var session = documentStore.OpenSession())
{
var docs = from gd in session.Query<GoldenDocument>()
from mf in gd.MergeMatchFields
from tf in mergeFields
where mf.Name == tf.Name
&& JaroWinklerCalculator.jaroWinkler(mf.Value, tf.Value) > .90d
&& !string.IsNullOrEmpty(mf.Value)
select gd;
}
I understand that ravenDB does not support SelectMany() so how would I go about getting the results from the Document store?
Create an index for this that would output the values you want to query on.
Note that you can't just execute arbitrary code the way you do here: JaroWinklerCalculator.jaroWinkler(mf.Value, tf.Value) > .90d
But you can use fuzzy queries, and they will do the same.

Speed up large Linq to Entities call MVC3

Below is the code I am using and the database table it is pulling from has about 92000 records in it. The way it is pulling right now it is pulling all 92000 records then doing the filtering.
What I am looking to do is the filtering on the initial pull from the DB so that it does not take aproximately 40 seconds to load the page.
This is something I am still new at so I am lost as to how to do this and make it work with my view
public ViewResult Makes()
{
var items = (from item in DBCacheHelper.recallslist
orderby item.MFGTXT ascending
select item.ToDomainRecall()).GroupBy(item => item.MFGTXT).Select(grp => grp.First());
return View(items);
}
public static IEnumerable<Recall> recallslist
{
get
{
if (c["GetAllRecalls"] == null)
{
c.Insert("GetAllRecalls", GetAllRecalls());
return (IEnumerable<Recall>)c["GetAllRecalls"];
}
else
{
return (IEnumerable<Recall>)c["GetAllRecalls"];
}
}
}
public static IEnumerable<Recall> GetAllRecalls()
{
using (DealerContext context = new DealerContext())
{
var items = from item in context.recalls.ToList<Recall>()
select item.ToDomainRecall();
return items.ToList<Recall>();
}
}
SELECT
[Extent1].[RecallsId] AS [RecallsId],
[Extent1].[RECORD_ID] AS [RECORD_ID],
[Extent1].[CAMPNO] AS [CAMPNO],
[Extent1].[MAKETXT] AS [MAKETXT],
[Extent1].[MODELTXT] AS [MODELTXT],
[Extent1].[YEARTXT] AS [YEARTXT],
[Extent1].[MFGCAMPNO] AS [MFGCAMPNO],
[Extent1].[COMPNAME] AS [COMPNAME],
[Extent1].[MFGNAME] AS [MFGNAME],
[Extent1].[BGMAN] AS [BGMAN],
[Extent1].[ENDMAN] AS [ENDMAN],
[Extent1].[RCLTYPECD] AS [RCLTYPECD],
[Extent1].[POTAFF] AS [POTAFF],
[Extent1].[ODATE] AS [ODATE],
[Extent1].[INFLUENCED_BY] AS [INFLUENCED_BY],
[Extent1].[MFGTXT] AS [MFGTXT],
[Extent1].[RCDATE] AS [RCDATE],
[Extent1].[DATEA] AS [DATEA],
[Extent1].[RPNO] AS [RPNO],
[Extent1].[FMVSS] AS [FMVSS],
[Extent1].[DESC_DEFECT] AS [DESC_DEFECT],
[Extent1].[CONEQUENCE_DEFECT] AS [CONEQUENCE_DEFECT],
[Extent1].[CORRECTIVE_ACTION] AS [CORRECTIVE_ACTION],
[Extent1].[NOTES] AS [NOTES],
[Extent1].[RCL_CMPT_ID] AS [RCL_CMPT_ID]
FROM [dbo].[Recalls] AS [Extent1]
Update:
Ultimately I would like to only pull records from the Recalls Table where the MFGTXT is equal to the
MakeName in the AutoMake Table
public class AutoMake
{
[Key]
public int MakeID { get; set; }
public string MakeName { get; set; }
public AutoMake ToDomainAutoMakes()
{
return new AutoMake
{
MakeID = this.MakeID,
MakeName = this.MakeName
};
}
}
public class Recall
{
[Key]
public int RecallsId { get; set; }
public string RECORD_ID { get; set; }
public string CAMPNO { get; set; }
public string MAKETXT { get; set; }
public string MODELTXT { get; set; }
public string YEARTXT { get; set; }
public string MFGCAMPNO { get; set; }
public string COMPNAME { get; set; }
public string MFGNAME { get; set; }
public string BGMAN { get; set; }
public string ENDMAN { get; set; }
public string RCLTYPECD { get; set; }
public string POTAFF { get; set; }
public string ODATE { get; set; }
public string INFLUENCED_BY { get; set; }
public string MFGTXT { get; set; }
public string RCDATE { get; set; }
public string DATEA { get; set; }
public string RPNO { get; set; }
public string FMVSS { get; set; }
public string DESC_DEFECT { get; set; }
public string CONEQUENCE_DEFECT { get; set; }
public string CORRECTIVE_ACTION { get; set; }
public string NOTES { get; set; }
public string RCL_CMPT_ID { get; set; }
public Recall ToDomainRecall()
{
return new Recall
{
RECORD_ID = this.RECORD_ID,
CAMPNO = this.CAMPNO,
MAKETXT = this.MAKETXT,
MODELTXT = this.MODELTXT,
YEARTXT = this.YEARTXT,
MFGCAMPNO = this.MFGCAMPNO,
COMPNAME = this.COMPNAME,
MFGNAME = this.MFGNAME,
BGMAN = this.BGMAN,
ENDMAN = this.ENDMAN,
RCLTYPECD = this.RCLTYPECD,
POTAFF = this.POTAFF,
ODATE = this.ODATE,
INFLUENCED_BY = this.INFLUENCED_BY,
MFGTXT = this.MFGTXT,
RCDATE = this.RCDATE,
DATEA = this.DATEA,
RPNO = this.RPNO,
FMVSS = this.FMVSS,
DESC_DEFECT = this.DESC_DEFECT,
CONEQUENCE_DEFECT = this.CONEQUENCE_DEFECT,
CORRECTIVE_ACTION = this.CORRECTIVE_ACTION,
NOTES = this.NOTES,
RCL_CMPT_ID = this.RCL_CMPT_ID
};
}
}
If you want to add server side filtering outside of your repository methods, you need to return your types as IQueryable rather than IEnumerable and not call .ToList, .AsEnumerable, or any other method that would cause .GetEnumerator to be called. Additionally, your cast `(IEnumerable)c["GetAllRecalls"];' forces LINQ to Objects to be used for subsequent requests rather than retaining the expression tree and using Entity Framework. That being said, you may need to move your call to ToDomainRecall method to after the additional filter is applied as well because that can't be translated to your database. Here are some of the changes you would need to make:
public ViewResult Makes()
{
var items = (from item in DBCacheHelper.recallslist
orderby item.MFGTXT ascending
select item.ToDomainRecall()).GroupBy(item => item.MFGTXT).Select(grp => grp.First());
return View(items);
}
public static IQueryable<Recall> recallslist
{
get
{
if (c["GetAllRecalls"] == null)
{
c.Insert("GetAllRecalls", GetAllRecalls(context));
}
return c["GetAllRecalls"];
}
}
public static IQueryable<Recall> GetAllRecalls(DealerContext context)
{
var items = context.recalls;
return items;
}
Looks like your DBaccess is done in the call to DBCacheHelper.recallslist.
You need to edit the sql that runs from/in this function.
As Eranga pointed out, you don't show how you are filtering the large number down to a smaller number of records. I assume you want 20 or 100 at a time? If so, please see the accepted answer here:
efficient way to implement paging
Specifically, this part which shows how to only retrieve rows x to y (where x = #p0 + 1 AND y = #p0 + #p1):
SELECT [t1].[CodCity],
[t1].[CodCountry],
[t1].[CodRegion],
[t1].[Name],
[t1].[Code]
FROM (
SELECT ROW_NUMBER() OVER (
ORDER BY [t0].[CodCity],
[t0].[CodCountry],
[t0].[CodRegion],
[t0].[Name],
[t0].[Code]) AS [ROW_NUMBER],
[t0].[CodCity],
[t0].[CodCountry],
[t0].[CodRegion],
[t0].[Name],
[t0].[Code]
FROM [dbo].[MtCity] AS [t0]
) AS [t1]
WHERE [t1].[ROW_NUMBER] BETWEEN #p0 + 1 AND #p0 + #p1
ORDER BY [t1].[ROW_NUMBER]

How to Count items in nested collection / codefirst EntityFramework

I've got CodeFirst collection defined as defined below.
For any given EmailOwnerId, I want to count the number of EmailDetailAttachments records exist without actually downloading all the images themselves.
I know I can do something like
var emailsToView = (from data in db.EmailDetails.Include("EmailDetailAttachments")
where data.EmailAccount.EmailOwnerId = 999
select data).ToList();
int cnt = 0;
foreach (var email in emailsToView)
{
cnt += email.EmailDetailAttachments.Count();
}
but that means I've already downloaded all the bytes of images from my far away server.
Any suggestion would be appreciated.
public class EmailDetail
{
[Key]
[DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public int EmailOwnerId {get;set;}
public virtual ICollection<ImageDetail> EmailDetailAttachments { get; set; }
..
}
public class ImageDetail
{
[Key]
[DatabaseGeneratedAttribute(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
[MaxLengthAttribute(256)]
public string FileName { get; set; }
[MaxLengthAttribute(256)]
public string ContentMimeType { get; set; }
public byte[] ImageDataBytes { get; set; }
public DateTime ImageCreation { get; set; }
}
The engine should be able to update this to a COUNT(*) statement.
var emailsToView = (from data in db.EmailDetails // no Include
where data.EmailAccount.EmailOwnerId = 999
select new {
Detail = data,
Count=data.EmailDetailAttachments.Count() }
).ToList();
But you'll have to verify if this produces the right (and more efficient) SQL.

Resources