EF5 (entity framework) memory leak and doesn't release after dispose - asp.net-web-api

So I'm using web api to expose data services. Initially I created my dbcontext as a static memory, and each time I open up my project under IISExpress, the memory balloons to over 100MB in memory. I understand that it isnt recommended to use static due to the solved answer here:
Entity framework context as static
So I went ahead and converted my application to using regular non-static dbcontext and included a dispose method on my api:
protected override void Dispose(Boolean disposing)
{
if (provider.Context != null)
{
provider.Context.Dispose();
provider = null;
}
base.Dispose(disposing);
}
Now every time I make a call, it goes through this method and disposes. Now, I open the application, still balloons to 100k, and each time I make a call, I watch the memory of my iisexpress process, and it keeps on going up and it's not coming back down after the dispose, it keeps increasing to almost 200MB+.
So static or not, memory explodes whenever I use it.
Initially I thought it was my web api that was causing it, until I removed all my services and just created the EF object in my api (I'm using breezejs, so this code is trivial, the actual implementation is down below, but makes no diff to memory consumption):
private DistributorLocationEntities context = new DistributorLocationEntities();
And bam, 110MB immediately.
Is there any helpful tips and tweaks on how I can release memory when I use it? Should I add garbage collect to my dispose()? Any pitfalls to allocating and deallocating memory rapidly like that? For example, I make calls to the service each time I make a keystroke to accomplish an "autocomplete" feature.
I'm also not certain what will happen if I put this in production, and we have dozens of users accessing the db; I wouldn't want the users to increase the memory to 1 or 2GB and it doesn't get released.
Side note: All my data services for now are searches, so there are no save changes or updates, though there can be later on though. Also, I don't return any linq queries as an array or enumerable, they remain as queryables throughout the service call.
One more thing, I do use breezejs, so I wrap up my context as such:
readonly EFContextProvider<DistributorLocationEntities> provider = new EFContextProvider<DistributorLocationEntities>();
and the tidbits that goes along with this:
Doc for Breeze's EFContextProvider
proxycreationenabled = false
ladyloadingenabled = false
idispose is not included
but I still dispose the context anyways, which makes no difference.

I don't know what you're doing. I do know that you should not have any static resources of any kind in your Web API controllers (breeze-flavored or not).
I strongly suspect you've violated that rule.
Adding a Dispose method no difference if the object is never disposed ... which it won't be if it is held in a static variable.
I do not believe that Breeze has any role in your problem whatsoever. You've already shown that it doesn't.
I suggest you start from a clean slate, forget Breeze for now, a get a simple Web API controller that creates a DbContext per request. When you've figured that out, proceed to add some Breeze.

As mentioned in Ward's comment, statics are a big no-no, so I spent time on moving my EF objects out of static. Dispose method didn't really help either.
I gave this article a good read:
http://msdn.microsoft.com/en-us/data/hh949853.aspx
There are quite a few performance options EF provides (that doesn't come out of the box). So here are a few things I've done:
Added pre-generated views to EF: T4 templates for generating views for EF4/EF5. The nice thing about this is that it abstracts away from the DB and pre-generates the view to decrease model load time
Next, I read this post on Contains in EF: Why does the Contains() operator degrade Entity Framework's performance so dramatically?. Apparently I saw an an attractive answer of converting my IEnumerable.Contains into a HashSet.Contains. This boosted my performance considerably.
Finally, reading the microsoft article, I realized there is a "AsNoTracking()" that you can hook up to the DBContext, this turns of automatic caching for that specific context in linq. So you can do something like this
var query = (from t in db.Context.Table1.AsNoTracking() select new { ... }
Something I didn't have to worry about was compiling queries in EF5, since it does it for you automatically, so you don't have to add CompileQuery.Compile(). Also if you're using EF 6 alpha 2, you don't need to worry about Contains or pre-generating views, since this is fixed in that version.
So when I start up my EF, this is a "cold" query execution, my memory goes high, but after recycling IIS, memory is cut in half and uses "warm" query execution. So that explains a lot!

Related

UWP: How to properly dispose a page / releasing memory efficiently

I am really curious to get to know how to clear/releaase/dispose a page properly.
Many answers were given such as
You do not, the garbage collector does it for you when appropriate.
How to dispose current page in UWP
foreach(var item in rootFrame.BackStack.ToList())
rootFrame.BackStack.Remove(item)
https://social.msdn.microsoft.com/Forums/exchange/en-US/2584e99b-3047-4d68-b22c-dcefc3ef9b83/uwpcpage-doesnt-destroyunload-itself-after-onnavigatedfrom?forum=wpdevelop
Also, to minimize the memory allocation, you must override method
OnNavigatedTo and OnNavigatedFrom.
In OnNavigatedTo method:
Instantiate all memory intensive object or resources
Add all events handlers you need
starts all timers, tasks or threads
In OnNavigatedFrom:
Dispose all resources
Stops all timers, tasks or threads
Remove references from all heavy objects
Remove all events handlers
Only if you really need, call GC.Collect() ove all events handlers Only if
you really need, call GC.Collect()
UWP Windows 10 App memory increasing on navigation
So, in our current project, I´ve just made a test to find out how I can "release" ressources.
The short answer is... I couldn´t.
I´ve set the NavigationCacheMode to disabled on purpose.
protected override void OnNavigatedFrom(NavigationEventArgs e) {
((ActivityViewModel)this.DataContext).OnDestroy();
this.DataContext = null;
base.OnNavigatedFrom(e);
}
I´ve actively called the OnDestroy method to my viewmodel (which is absolutely not great), to set heavy objects to null and release events
public void OnDestroy() {
this.ActivitySource.FetchRows -= this.ActivitySource_FetchRows;
this.ActivitySource.GetUniqueValues -= this.ActivitySource_GetUniqueValues;
this.TaskObjectSource = null;
this.UnitOfWork.Dispose();
this.ActivitySource = null;
this.NavigateToTaskCommand = null;
this.SelectedItem = null;
}
Guess what. Each time I´ve reentered the page, the ram will slightly increase. If you would repeat this for x times, your application will unevitable throw OutOfMemoryException.
So I was thinking about this and came up with the conclusion that a page is just a control container and should be cached, whereas your datacontext holds the data information which drives the ui controls through its bindings. In the end, it makes most sense to me. Why do create a new instance of a page when you can just exchange its datacontext or manipulate properties in its given context. But then, why is the default NavigationCacheMode 'Disabled'? How would Dialogs be managed due to its volatile behavior? The more I think about this, the more questions I have.
How does "the big guys" dealing with this "issue"? Maybe my understanding is wrong, but then there might be sources to get knowledge.
From my current perspective, I really enjoy coding with the uwp. But sometimes it makes me feel like certain things are not well optimized or missing (On-Premise Active Directory :/ ).
If you need additional Information, or even a reproduced sample, feel free to ask.
Thank you in advance.
Best regards

Magento register models for performance gain

I have a large application in Magento which is pretty "heavy" in terms of data collections and actions that are performed. Currently I'm trying to optimize the performance and I noticed that during a regular page load, around 900 Mage::getModel calls are being performed. The call itself is quite fast but when 900 calls are made, this affect performance as all the calls take around 2 seconds.
I was wondering if it's safe to use Magento's registry functionality to register models that have no construct arguments. Basically if a Mage::getModel('sales/quote') is called, after loading the class I intend to register the instance under a unique key (like 'model_register_'.className) and all subsequent Mage::getModel('sales/quote') calls will no longer create a new instance of the model but return the one in the registry, which should improve performance. This, of course, would only be used for calls that have no $constructArguments in the Mage::getModel call.
Has anyone done this before? As I am interested if this approach is safe or if this might cause other issues.
Apparently this does not work because registry keeps a reference to the object. So when retrieving the model from the registry, you would not get a "clean" instance

Win from application gets slow

I have built an application using Visual Studio .NET and it works fine. After the application is used for more than 2-3 hours it starts to get slow and I don't know why. I have used GC.Collect(); to get memory leak problems but now I have the new one.
Does anyone know a solution?
If you really have a memory leak, just calling GC.Collect() will get you nowhere. The GarbageCollector can only collect those objects, that are not referenced from others anymore.
If you do not cleanup your objects properly, the GC will not collect anything.
When handling with memory consumptions, you should strongly consider the following patterns:
Weak Events (MSDN Documentation here)
If you do not unsubscribe from events, the subscribing objects will never be released into the Garbage Collection. GC.Collect() will NOT remove those objects and they will clutter your memory.
Implement the IDisposable interface (MSDN documentation here)
(I strongly suggest to read this ducumentation as I have seen lots of wrong implementations.)
You should always free resources that you used. Call Dispose() on every object that offers it!
The same applies to streams. Always call Close() on every object that offers this.
To make points 2. and 3. easier you can use the using blocks. (MSDN documentation here)
As soon as these code blocks go out of scope they automatically call the appropriate Dispose() or Close() methods on the given object. This is the same, but more convinient, as using a try... finally combination.
Try a memory profiler, such as the ANTS Memory Profiler. First you need to understand what's going on, then you can think about how to fix it.
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

Cache Management with Numerous Similar Database Queries

I'm trying to introduce caching into an existing server application because the database is starting to become overloaded.
Like many server applications we have the concept of a data layer. This data layer has many different methods that return domain model objects. For example, we have an employee data access object with methods like:
findEmployeesForAccount(long accountId)
findEmployeesWorkingInDepartment(long accountId, long departmentId)
findEmployeesBySearch(long accountId, String search)
Each method queries the database and returns a list of Employee domain objects.
Obviously, we want to try and cache as much as possible to limit the number of queries hitting the database, but how would we go about doing that?
I see a couple possible solutions:
1) We create a cache for each method call. E.g. for findEmployeesForAccount we would add an entry with a key account-employees-accountId. For findEmployeesWorkingInDepartment we could add an entry with a key department-employees-accountId-departmentId and so on. The problem I see with this is when we add a new employee into the system, we need to ensure that we add it to every list where appropriate, which seems hard to maintain and bug-prone.
2) We create a more generic query for findEmployeesForAccount (with more joins and/or queries because more information will be required). For other methods, we use findEmployeesForAccount and remove entries from the list that don't fit the specified criteria.
I'm new to caching so I'm wondering what strategies people use to handle situations like this? Any advice and/or resources on this type of stuff would be greatly appreciated.
I've been struggling with the same question myself for a few weeks now... so consider this a half-answer at best. One bit of advice that has been working out well for me is to use the Decorator Pattern to implement the cache layer. For example, here is an article detailing this in C#:
http://stevesmithblog.com/blog/building-a-cachedrepository-via-strategy-pattern/
This allows you to literally "wrap" your existing data access methods without touching them. It also makes it very easy to swap out the cached version of your DAL for the direct access version at runtime quite easily (which can be useful for unit testing).
I'm still struggling to manage my cache keys, which seem to spiral out of control when there are numerous parameters involved. Inevitably, something ends up not being properly cleared from the cache and I have to resort to heavy-handed ClearAll() approaches that just wipe out everything. If you find a solution for cache key management, I would be interested, but I hope the decorator pattern layer approach is helpful.

subsonic with support for caching

Having a project with following requirements in mind.
data reading intensive application.
100 max concurrent users a times. Application have very high traffic
Though data is huge it is getting modified only once a day
Decided to use subsonic cause of ease of development and potential to work in high traffic environment.
Though few things are not yet found/solved to work with SubSonic 3
Which type of layer to use Active Records, Repository, Linq To SQL
working with paging / sorting stored procedures (cause they will give better performance over inbuilt paging mechanism, when displaying 10000+ rows with paging and sorting. right?? )
Caching, with project requirement it is quite clear, heavy use of caching is required. But could not find suitable solution, which will work with subsonic.
do I have to create separate layer for it and if yes, a short example would be helpful.
I wrote a CacheUtil class for subsonic 2.x ActiveRecord. It's based on some code someone posted on the old subsonic forums. (This is from a forum that was deleted before the last forum was removed. This is why software forums should be permanent.) Here is an example of a cache Find method. You could adapt it to ss3. There are also inserts, fetchall, delete, clear, etc. Rob Connery said at the time that caching was problematic, and it was left out of ss2 on purpose. By using HttpRuntime.Cache I share the cache between a web application and service simultaneously. I believe I can do this since it's a small application, always on a single server.
public static RecordBase<T> Find<T, ListType>(object primaryKeyValue)
where T: RecordBase<T>, new()
where ListType: AbstractList<T, ListType>, new()
{
string key = typeof(T).ToString();
if(HttpRuntime.Cache[key] == null)
FetchAll<T, ListType>();
if(HttpRuntime.Cache[key] != null)
{
ListType collection = (ListType)HttpRuntime.Cache[key];
foreach(T item in collection)
{
if(item.GetPrimaryKeyValue().Equals(primaryKeyValue))
return item;
}
}
return null;
}
I wrote a post about how I used caching with SubSonic 2.x. It isn't 100% compatible with 3.x but the concepts are the same.
I answered this similarly over here Thread-safe cache libraries for .NET. Basically you need a CollectionCacheManager - I then add a layer on top for each type and funnel all requests through this individual cache controllers, which in turn are using the 1 collectioncachecontroller. At the outer layer I mix pure subsonic, linq, whatever fits the bill at the time. That's the beauty of SubSonic is that it should not get in your way. As far as stored proc performance I would point to Jeff Atwoods articles over at CodingHorror and reevaulaute your savings. Hardware is dirt cheap, as is memory, databases are not. Personally I keep the database super simple and lightweight, and prefer to let my webserver cache everything in memory. The database server gets to do very little work which is the way I like it. Adding a few extra load balanced web servers isn't nearly as big of a deal as increasing database throughput, clustering, or sharding a a DB. SQL & Stored Procs can also be ridiculously difficult to write, and maintain. Take that budget that you would have spent on your time doing that, and instead beef up your hardware... Remember hardware is dirt cheap, good developers are not. Good luck!

Resources