So I'm using web api to expose data services. Initially I created my dbcontext as a static memory, and each time I open up my project under IISExpress, the memory balloons to over 100MB in memory. I understand that it isnt recommended to use static due to the solved answer here:
Entity framework context as static
So I went ahead and converted my application to using regular non-static dbcontext and included a dispose method on my api:
protected override void Dispose(Boolean disposing)
{
if (provider.Context != null)
{
provider.Context.Dispose();
provider = null;
}
base.Dispose(disposing);
}
Now every time I make a call, it goes through this method and disposes. Now, I open the application, still balloons to 100k, and each time I make a call, I watch the memory of my iisexpress process, and it keeps on going up and it's not coming back down after the dispose, it keeps increasing to almost 200MB+.
So static or not, memory explodes whenever I use it.
Initially I thought it was my web api that was causing it, until I removed all my services and just created the EF object in my api (I'm using breezejs, so this code is trivial, the actual implementation is down below, but makes no diff to memory consumption):
private DistributorLocationEntities context = new DistributorLocationEntities();
And bam, 110MB immediately.
Is there any helpful tips and tweaks on how I can release memory when I use it? Should I add garbage collect to my dispose()? Any pitfalls to allocating and deallocating memory rapidly like that? For example, I make calls to the service each time I make a keystroke to accomplish an "autocomplete" feature.
I'm also not certain what will happen if I put this in production, and we have dozens of users accessing the db; I wouldn't want the users to increase the memory to 1 or 2GB and it doesn't get released.
Side note: All my data services for now are searches, so there are no save changes or updates, though there can be later on though. Also, I don't return any linq queries as an array or enumerable, they remain as queryables throughout the service call.
One more thing, I do use breezejs, so I wrap up my context as such:
readonly EFContextProvider<DistributorLocationEntities> provider = new EFContextProvider<DistributorLocationEntities>();
and the tidbits that goes along with this:
Doc for Breeze's EFContextProvider
proxycreationenabled = false
ladyloadingenabled = false
idispose is not included
but I still dispose the context anyways, which makes no difference.
I don't know what you're doing. I do know that you should not have any static resources of any kind in your Web API controllers (breeze-flavored or not).
I strongly suspect you've violated that rule.
Adding a Dispose method no difference if the object is never disposed ... which it won't be if it is held in a static variable.
I do not believe that Breeze has any role in your problem whatsoever. You've already shown that it doesn't.
I suggest you start from a clean slate, forget Breeze for now, a get a simple Web API controller that creates a DbContext per request. When you've figured that out, proceed to add some Breeze.
As mentioned in Ward's comment, statics are a big no-no, so I spent time on moving my EF objects out of static. Dispose method didn't really help either.
I gave this article a good read:
http://msdn.microsoft.com/en-us/data/hh949853.aspx
There are quite a few performance options EF provides (that doesn't come out of the box). So here are a few things I've done:
Added pre-generated views to EF: T4 templates for generating views for EF4/EF5. The nice thing about this is that it abstracts away from the DB and pre-generates the view to decrease model load time
Next, I read this post on Contains in EF: Why does the Contains() operator degrade Entity Framework's performance so dramatically?. Apparently I saw an an attractive answer of converting my IEnumerable.Contains into a HashSet.Contains. This boosted my performance considerably.
Finally, reading the microsoft article, I realized there is a "AsNoTracking()" that you can hook up to the DBContext, this turns of automatic caching for that specific context in linq. So you can do something like this
var query = (from t in db.Context.Table1.AsNoTracking() select new { ... }
Something I didn't have to worry about was compiling queries in EF5, since it does it for you automatically, so you don't have to add CompileQuery.Compile(). Also if you're using EF 6 alpha 2, you don't need to worry about Contains or pre-generating views, since this is fixed in that version.
So when I start up my EF, this is a "cold" query execution, my memory goes high, but after recycling IIS, memory is cut in half and uses "warm" query execution. So that explains a lot!
I have such problem with my application that´s running in real time.
It´s getting values an plotting two graphs. Also is updating a few (8) QLabels and displaying them in window. The application should run for a very long time and processing data so I wanted to test it in faster regime and I set the sending values to the application for each 10ms. When I start the application everything works fine, but after some time QLabels stuck or plotting graphs stuck and in the end application crashes.
I would like to ask if 10ms is too fast for updating QLabels and graphs (QCustomPlot) and that is the reason of crashing the program or I need to search for the problem somewhere else?
void mainWnd::useDataFromPipe(double val) {
if(val<=*DOF) {
FSOcounter=FSOcounter+1.0;
lastRecievedFSO->saveValue(val);
updateGraphMutex->lock();
FSOvectorXshort->replace(1, FSOcounter);
updateGraphMutex->unlock();
if(!firstDataFSO()) {
if(val>maximumRecievedFSO->loadValue()) maximumRecievedFSO->saveValue(val);
else if(val<minimumRecievedFSO->loadValue()) minimumRecievedFSO->saveValue(val);
}
else {
maximumRecievedFSO->saveValue(val);
minimumRecievedFSO->saveValue(val);
}
numberOfRecievedValuesFSO->saveValue(numberOfRecievedValuesFSO->loadValue()+1.0);
}
else {
RFcounter=RFcounter+1.0;
lastRecievedRF->saveValue(val);
updateGraphMutex->lock();
RFvectorXshort->replace(1, RFcounter);
updateGraphMutex->unlock();
if(!firstDataRF()) {
if(val>maximumRecievedRF->loadValue()) maximumRecievedRF->saveValue(val);
else if(val<minimumRecievedRF->loadValue()) minimumRecievedRF->saveValue(val);
}
else {
maximumRecievedRF->saveValue(val);
minimumRecievedRF->saveValue(val);
}
numberOfRecievedValuesRF->saveValue(numberOfRecievedValuesRF->loadValue()+1.0);
}
}
where saveValue() and loadValue() are
void infoLine::saveValue(double val) {
*value=val;
valueLabel->setText(QString("<b>%1</b>").arg(val));
}
double infoLine::loadValue(void) {
return *value;
}
You're not "updating" the labels every 10ms, you're merely calling setText or similar methods on them every 10ms. The labels will be updated as dictated by how many events are in the event queue. As long as you "update" the labels by calling setText etc., you're OK - this should not be a problem.
A crash is usually due to a memory bug. A memory bug typically can be due to running out of memory, a double free, or access to deallocated memory.
What you describe seems like a memory bug due to running out of memory. The most trivial explanation is that you're leaking memory. That's an easy one - smart pointers (QScopedPointer and QSharedPointer) will help with that.
You may also not be leaking memory, but recursing too deeply if you indeed recurse into the event loop.
You may also be updating the widget(s) incorrectly, resulting in forced, repeated, expensive repaints that are not compressed into one repaint.
You'd need to show how you "update" the plot. While the labels are implemented correctly, and repeated calls to various QLabel setXxxx methods are cheap, the custom plot class may simply be implemented incorrectly.
The idiomatic, and indeed the only correct way of updating a widget in Qt is:
Set some data members.
Call update().
This is what QLabel::setText does internally, and it is what any other widget should be doing as well. This functionality should be exposed in a setXxxx-like method, and is an implementation detail, although an important one. The users of the widget don't need to be aware of it.
The update events are posted to the widget and are compressed, such that repeated updates without returning to the event loop result in only one repaint.
It's really hard to tell without seeing a self-contained, minimal example. By minimal I mean nothing that has no bearing on the issue should be left in.
I am working with mono droid, using the mvvmcross framework provided by slodge. However I am having some memory issues. I am disposing bitmaps in the activities ondestroy methods and I am wondering if it is possible to help the GC collecting unused objects of viewmodels. If you try setting the viewmodel in the activity to null it all goes to hell and it is clearly not the right way to go.
Do you guys have any suggestions to an approach?
Regards
The mvx framework tries to ensure that the activity owns the viewmodel.
So in theory, after your activity has being destroyed, then the gc should be able to collect all of your c# objects - the activity, the views it owns, the view model and the objects it owns.
Where i've seen this this go wrong is where any 'global' or singleton object owns a reference to a view or viewmodel object. For example:
if a view registers itself with a singleton - eg an http image loader - and then that singleton keeps a reference to the view, preventing it from being garbage collected.
if a viewmodel subscribes to an event on a central service (often a singleton) and doesn't unsubscribe from it - then in this situation, the viewmodel can't be garbage collected (and often this also prevents other objects being collected too)
Generally both these types of errors can be solved by performing cleanup actions on activity destroy. However, other approaches are also available - eg for event subscriptions you can try using weak references (this is an approach taken on other platforms too - eg mvvm light's messenger)
From experience, the areas where leaks are most noticeable are around 'big objects' like images - their size helps them become noticeable. However, the real challenge on monodroid is identifying where the leaks are - fixing them is generally comparatively easy.
Sadly, there isn't currently a memory profiler available for droid. If you are cross-compiling to wp7, then certainly for viewmodel objects/leaks you can use its memory profiler. If not, then the way I generally try to solve memory leaks is to amplify them - try writing a sample that rapidly reproduces them - eg by adding large byte[] members to data elements or by rapidly repeating actions. Once you have the leak easily reproduced, then you can try to find the leaks by placing trace statements in finalizers, in event remove handlers, etc.
I have built an application using Visual Studio .NET and it works fine. After the application is used for more than 2-3 hours it starts to get slow and I don't know why. I have used GC.Collect(); to get memory leak problems but now I have the new one.
Does anyone know a solution?
If you really have a memory leak, just calling GC.Collect() will get you nowhere. The GarbageCollector can only collect those objects, that are not referenced from others anymore.
If you do not cleanup your objects properly, the GC will not collect anything.
When handling with memory consumptions, you should strongly consider the following patterns:
Weak Events (MSDN Documentation here)
If you do not unsubscribe from events, the subscribing objects will never be released into the Garbage Collection. GC.Collect() will NOT remove those objects and they will clutter your memory.
Implement the IDisposable interface (MSDN documentation here)
(I strongly suggest to read this ducumentation as I have seen lots of wrong implementations.)
You should always free resources that you used. Call Dispose() on every object that offers it!
The same applies to streams. Always call Close() on every object that offers this.
To make points 2. and 3. easier you can use the using blocks. (MSDN documentation here)
As soon as these code blocks go out of scope they automatically call the appropriate Dispose() or Close() methods on the given object. This is the same, but more convinient, as using a try... finally combination.
Try a memory profiler, such as the ANTS Memory Profiler. First you need to understand what's going on, then you can think about how to fix it.
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/
DataSet and DataTable both implement IDisposable, so, by conventional best practices, I should call their Dispose() methods.
However, from what I've read so far, DataSet and DataTable don't actually have any unmanaged resources, so Dispose() doesn't actually do much.
Plus, I can't just use using(DataSet myDataSet...) because DataSet has a collection of DataTables.
So, to be safe, I'd need to iterate through myDataSet.Tables, dispose of each of the DataTables, then dispose of the DataSet.
So, is it worth the hassle to call Dispose() on all of my DataSets and DataTables?
Addendum:
For those of you who think that DataSet should be disposed:
In general, the pattern for disposing is to use using or try..finally, because you want to guarantee that Dispose() will be called.
However, this gets ugly real fast for a collection. For example, what do you do if one of the calls to Dispose() thrown an exception? Do you swallow it (which is "bad") so that you can continue on to dispose the next element?
Or, do you suggest that I just call myDataSet.Dispose(), and forget about disposing the DataTables in myDataSet.Tables?
Here are a couple of discussions explaining why Dispose is not necessary for a DataSet.
To Dispose or Not to Dispose ?:
The Dispose method in DataSet exists ONLY because of side effect of inheritance-- in other words, it doesn't actually do anything useful in the finalization.
Should Dispose be called on DataTable and DataSet objects? includes some explanation from an MVP:
The system.data namespace (ADONET) does not contain
unmanaged resources. Therefore there is no need to dispose any of those as
long as you have not added yourself something special to it.
Understanding the Dispose method and datasets? has a with comment from authority Scott Allen:
In pratice we rarely Dispose a DataSet because it offers little benefit"
So, the consensus there is that there is currently no good reason to call Dispose on a DataSet.
Update (December 1, 2009):
I'd like to amend this answer and concede that the original answer was flawed.
The original analysis does apply to objects that require finalization – and the point that practices shouldn’t be accepted on the surface without an accurate, in-depth understanding still stands.
However, it turns out that DataSets, DataViews, DataTables suppress finalization in their constructors – this is why calling Dispose() on them explicitly does nothing.
Presumably, this happens because they don’t have unmanaged resources; so despite the fact that MarshalByValueComponent makes allowances for unmanaged resources, these particular implementations don’t have the need and can therefore forgo finalization.
(That .NET authors would take care to suppress finalization on the very types that normally occupy the most memory speaks to the importance of this practice in general for finalizable types.)
Notwithstanding, that these details are still under-documented since the inception of the .NET Framework (almost 8 years ago) is pretty surprising (that you’re essentially left to your own devices to sift though conflicting, ambiguous material to put the pieces together is frustrating at times but does provide a more complete understanding of the framework we rely on everyday).
After lots of reading, here’s my understanding:
If an object requires finalization, it could occupy memory longer than it needs to – here’s why: a) Any type that defines a destructor (or inherits from a type that defines a destructor) is considered finalizable; b) On allocation (before the constructor runs), a pointer is placed on the Finalization queue; c) A finalizable object normally requires 2 collections to be reclaimed (instead of the standard 1); d) Suppressing finalization doesn’t remove an object from the finalization queue (as reported by !FinalizeQueue in SOS)
This command is misleading; Knowing what objects are on the finalization queue (in and of itself) isn’t helpful; Knowing what objects are on the finalization queue and still require finalization would be helpful (is there a command for this?)
Suppressing finalization turns a bit off in the object's header indicating to the runtime that it doesn’t need to have its Finalizer invoked (doesn’t need to move the FReachable queue); It remains on the Finalization queue (and continues to be reported by !FinalizeQueue in SOS)
The DataTable, DataSet, DataView classes are all rooted at MarshalByValueComponent, a finalizable object that can (potentially) handle unmanaged resources
Because DataTable, DataSet, DataView don’t introduce unmanaged resources, they suppress finalization in their constructors
While this is an unusual pattern, it frees the caller from having to worry about calling Dispose after use
This, and the fact that DataTables can potentially be shared across different DataSets, is likely why DataSets don’t care to dispose child DataTables
This also means that these objects will appear under the !FinalizeQueue in SOS
However, these objects should still be reclaimable after a single collection, like their non-finalizable counterparts
4 (new references):
http://www.devnewsgroups.net/dotnetframework/t19821-finalize-queue-windbg-sos.aspx
http://blogs.msdn.com/tom/archive/2008/04/28/asp-net-tips-looking-at-the-finalization-queue.aspx
http://issuu.com/arifaat/docs/asp_net_3.5unleashed
http://msdn.microsoft.com/en-us/magazine/bb985013.aspx
http://blogs.msdn.com/tess/archive/2006/03/27/561715.aspx
Original Answer:
There are a lot of misleading and generally very poor answers on this - anyone who's landed here should ignore the noise and read the references below carefully.
Without a doubt, Dispose should be called on any Finalizable objects.
DataTables are Finalizable.
Calling Dispose significantly speeds up the reclaiming of memory.
MarshalByValueComponent calls GC.SuppressFinalize(this) in its Dispose() - skipping this means having to wait for dozens if not hundreds of Gen0 collections before memory is reclaimed:
With this basic understanding of finalization we
can already deduce some very important
things:
First, objects that need finalization
live longer than objects that do not.
In fact, they can live a lot longer.
For instance, suppose an object that
is in gen2 needs to be finalized.
Finalization will be scheduled but the
object is still in gen2, so it will
not be re-collected until the next
gen2 collection happens. That could be
a very long time indeed, and, in fact,
if things are going well it will be a
long time, because gen2 collections
are costly and thus we want them to
happen very infrequently. Older
objects needing finalization might
have to wait for dozens if not
hundreds of gen0 collections before
their space is reclaimed.
Second, objects that need finalization
cause collateral damage. Since the
internal object pointers must remain
valid, not only will the objects
directly needing finalization linger
in memory but everything the object
refers to, directly and indirectly,
will also remain in memory. If a huge
tree of objects was anchored by a
single object that required
finalization, then the entire tree
would linger, potentially for a long
time as we just discussed. It is
therefore important to use finalizers
sparingly and place them on objects
that have as few internal object
pointers as possible. In the tree
example I just gave, you can easily
avoid the problem by moving the
resources in need of finalization to a
separate object and keeping a
reference to that object in the root
of the tree. With that modest change
only the one object (hopefully a nice
small object) would linger and the
finalization cost is minimized.
Finally, objects needing finalization
create work for the finalizer thread.
If your finalization process is a
complex one, the one and only
finalizer thread will be spending a
lot of time performing those steps,
which can cause a backlog of work and
therefore cause more objects to linger
waiting for finalization. Therefore,
it is vitally important that
finalizers do as little work as
possible. Remember also that although
all object pointers remain valid
during finalization, it might be the
case that those pointers lead to
objects that have already been
finalized and might therefore be less
than useful. It is generally safest to
avoid following object pointers in
finalization code even though the
pointers are valid. A safe, short
finalization code path is the best.
Take it from someone who's seen 100s of MBs of non-referenced DataTables in Gen2: this is hugely important and completely missed by the answers on this thread.
References:
1 -
http://msdn.microsoft.com/en-us/library/ms973837.aspx
2 -
http://vineetgupta.spaces.live.com/blog/cns!8DE4BDC896BEE1AD!1104.entry
http://www.dotnetfunda.com/articles/article524-net-best-practice-no-2-improve-garbage-collector-performance-using-finalizedispose-pattern.aspx
3 -
http://codeidol.com/csharp/net-framework/Inside-the-CLR/Automatic-Memory-Management/
You should assume it does something useful and call Dispose even if it does nothing in current .NET Framework incarnations. There's no guarantee it will stay that way in future versions leading to inefficient resource usage.
Even if an object has no unmanaged resources, disposing might help GC by breaking object graphs. In general, if an object implements IDisposable, Dispose() should be called.
Whether Dispose() actually does something or not depends on the given class. In case of DataSet, Dispose() implementation is inherited from MarshalByValueComponent. It removes itself from container and calls Disposed event. The source code is below (disassembled with .NET Reflector):
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
lock (this)
{
if ((this.site != null) && (this.site.Container != null))
{
this.site.Container.Remove(this);
}
if (this.events != null)
{
EventHandler handler = (EventHandler) this.events[EventDisposed];
if (handler != null)
{
handler(this, EventArgs.Empty);
}
}
}
}
}
Do you create the DataTables yourself? Because iterating through the children of any Object (as in DataSet.Tables) is usually not needed, as it's the job of the Parent to dispose all its child members.
Generally, the rule is: If you created it and it implements IDisposable, Dispose it. If you did NOT create it, then do NOT dispose it, that's the job of the parent object. But each object may have special rules, check the Documentation.
For .NET 3.5, it explicitly says "Dispose it when not using anymore", so that's what I would do.
I call dispose anytime an object implements IDisposeable. It's there for a reason.
DataSets can be huge memory hogs. The sooner they can be marked for clean up, the better.
update
It's been 5 years since I answered this question. I still agree with my answer. If there is a dispose method, it should be called when you are done with the object. The IDispose interface was implemented for a reason.
If your intention or the context of this question is really garbage collection, then you can set the datasets and datatables to null explicitly or use the keyword using and let them go out of scope. Dispose does not do much as Tetraneutron said it earlier. GC will collect dataset objects that are no longer referenced and also those that are out of scope.
I really wish SO forced people down voting to actually write a comment before downvoting the answer.
Datasets implement IDisposable thorough MarshalByValueComponent, which implements IDisposable. Since datasets are managed there is no real benefit to calling dispose.
Try to use Clear() function.
It works great for me for disposing.
DataTable dt = GetDataSchema();
//populate dt, do whatever...
dt.Clear();
No need to Dispose()
because DataSet inherit MarshalByValueComponent class and MarshalByValueComponent implement IDisposable Interface
This is the right way to properly Dispose the DataTable.
private DataTable CreateSchema_Table()
{
DataTable td = null;
try
{
td = new DataTable();
//use table DataTable here
return td.Copy();
}
catch { }
finally
{
if (td != null)
{
td.Constraints.Clear();
td.Clear();
td.Dispose();
td = null;
}
}
}
And this can be the best/proper way to Dispose and release the memory consumed by DataSet.
try
{
DataSet ds = new DataSet("DS");
//use table DataTable here
}
catch { }
finally
{
if (ds != null)
{
ds.EnforceConstraints = false;
ds.Relations.Clear();
int totalCount = ds.Tables.Count;
for (int i = totalCount - 1; i >= 0; i--)
{
DataTable td1 = ds.Tables[i];
if (td1 != null)
{
td1.Constraints.Clear();
td1.Clear();
td1.Dispose();
td1 = null;
}
}
ds.Tables.Clear();
ds.Dispose();
ds = null;
}
}