This question is not really related to any specific code or even language.
If you allocate huge (exceeding phisical memory) amount of memory on Windows it causes entire operating system to become fully unresponsive - including mouse cursor which typically was able to move even with entire system crashed.
Working Set API seem to not solve the problem - it seems that all applications have an initial max working set size already set to a rather low level.
I hoped memory mapped files (via boost api) would help OS make better decisions about page loading/unloading - but again even single pass trough large data freezes the system.
Are there any magic WinAPI calls or other good programming practice (other than manual management of entire commited memory and manual data caching in files) that would keep the operating system and other applications reasonably stable while using such huge amount of data?
My question is simple. I just can't seem to find much information on it googling around. If I set up Windows Error Reporting to create memory dumps when an IIS app pool crashes (See article here for rundown.) could it cause any noticeable degradation in performance in terms of IIS serving up apps and websites?
We are looking to set this up in production to assist in tracking down issues when an app pool crashes. Also, is something like this recommended for production servers?
I assume you're using the LocalDumps Registry key to achieve this.
That Registry key has a setting called DumpType where you can specify the amount of data being collected.
The time it takes to capture the crash dump will mainly depend on the amount of data to be written to disk. A full memory crash dump of IIS can easily take 4 GB, which may take 8 seconds to be written to disk at a 500 MB/s disk throughput rate.
During those 8 seconds, another instance of IIS may still serve pages from cached files very fast, but might have trouble serving files that need to be read from disk.
You could mitigate this a bit by spending an extra disk or different partition for the crash dumps. You'll then just have the memory, CPU and SATA overhead.
Anyway, you'll not leave this setting on for very long. Just capture a few crash dumps and then disable it. You'll make your customers happier if you resolve the crash. Therefore, the performance impact IMHO is acceptable.
If you want to know the exact impact, you'd need to set up a load test system, serve pages and implement a website that crashes. You'll then notice (or not) the performance degradation and you'll be able to measure it.
I have windows application I wrote.
I installed it on virtual server (vmware) that holds windows server 2008 and for some reason the application getting bigger and bigger. I used perfmon in order to see maybe there is a memory leak - but as I understand, there isn't:
Here is the proccess in task manager:
There are also two proccesses that use a lot of memory and cpu but are steady and not growing like SimeserManager.exe.
The memory growing slows the surfing on sited that this server holds.
Before this week I used the my application on phizical server with windows server 2003 and there were no problem with surfing. I can't restore the situtaion in the phizical machine since I don't have it anymore, but I don't believe there was memory error when using the phizical server.
The application is written in c# .net using visual studio 2010.
What can be the problem?
Where can I get some clues?
UPDATE
I get ANTS memory profiler and tried to seek for the problem. I created memory snapshots and here is the results:
Now I'm really lost.
I tryed the standart filters but didn't manage to find a clue for the problem. In the image you can see there is increase in the private bytes. Does that sais there is a memory leak?
Can anyone give me some clues how to continue?
Thanks!
We don't have enough information here to really debug your application. However, there are tools you can use to identify and solve this issue in your application. I would suggest you use the ANTS Memory Profiler from RedGate to help you look for your problem. Here is a link to it:
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/
It isn't free but it is cheap and extremely effective. Get the 14-day free trial and run it on your application. I would go as far as to say that if it doesn't find the issue, the issue probably isn't with your application.
As for the other processes that are taking a lot of memory, this is normal. SQL Server tries to get as much memory as possible. Running other applications on the same box as a SQL server may cause you performance issues if you aren't careful. Here is a good article on how SQL Server uses memory:
http://sqlblog.com/blogs/jonathan_kehayias/archive/2009/08/24/troubleshooting-the-sql-server-memory-leak-or-understanding-sql-server-memory-usage.aspx
As for IIS memory usage (the other process that was using lots of memory), there could be multiple reasons for this. I would suggest you read through this forum to get a better idea of what it could be (if it truly is an issue):
http://forums.iis.net/t/1150494.aspx
My site goes slow and stops access certain services externally if we check the Process monitor we see that it is normally due to the ‘w3p.exe’ process – which is the background process for running the website – it regularly reaches 99/100% - killing the process/restarting the WebPublishing service reolves tis – my webhost says this can only be due to bad coding ....can someone comment on this ??…
When performance testing a reasonably straightforward website (coded in ASP.Net) I saw it slow to a crawl with memory use going through the roof over time. Each time recycling the w3wp process restored performance back to normal.
I never got around to figuring out why (the load we were testing with was way above normal, and it could have been worked around anyway by recycling the w3wp service more frequently), but my bet would have been that it was viewstate causing the slowdown. A lot of pages had very large viewstate which wasnt being used in any way - I can fesably see how loading large viewstate values might cause memory related performance degregation over time.
What language is the site coded in? I recently ran into the same problem on a server running IIS6/PHP and found the following bug -
http://bugs.php.net/bug.php?id=37575
Upgrading PHP to 5.3 solved the issue.
I am debugging an application which slows down the system very badly. The application loads a large amount of data (some 1000 files each of half an MB) from the local hard disk.The files are loaded as memory mapped files and are mapped only when needed. This means that at any given point in time the virtual memory usage does not exceed 300 MB.
I also checked the Handle count using handle.exe from sysinternals and found that there are at the most some 8000 odd handles opened. When the data is unloaded it drops to around 400. There are no handle leaks after each load and unload operation.
After 2-3 Load unload cycles, during one load, the system becomes very slow. I checked the virtual memory usage of the application as well as the handle counts at this point and it was well within the limits (VM about 460MB not much fragmentation also, handle counts 3200).
I want how an application could make the system very slow to respond? What other tools can I use to debug this scenario?
Let me be more specific, when i mean system it is entire windows that is slowing down. Task manager itself takes 2 mins to come up and most often requires a hard reboot
The fact that the whole system slows downs is very annoying, it means you can not attach a profiler easily, it also means it would be even difficult to stop the profiling session in order to view the results ( since you said it require a hard reboot ).
The best tool suited for the job in this situation is ETW ( Event Tracing for Windows ), these tools are great, will give you the exact answer you are looking for
Check them out here
http://msdn.microsoft.com/en-us/library/cc305210.aspx
and
http://msdn.microsoft.com/en-us/library/cc305221.aspx
and
http://msdn.microsoft.com/en-us/performance/default.aspx
Hope this works.
Thanks
Tools you can use at this point:
Perfmon
Event Viewer
In my experience, when things happen to a system that prevent Task Manager from popping up, they're usually of the hardware variety -- checking the system event log of Event Viewer is sometimes just full of warnings or errors that some hardware device is timing out.
If Event Viewer doesn't indicate that any kind of loggable hardware error is causing the slowdown, then try Perfmon -- add counters for system objects to track file read, exceptions, context switches etc. per second and see if there's something obvious there.
Frankly the sort of behavior demonstrated is meant to be impossible - by design - for user-mode code to cause. WinNT goes to a lot of effort to insulate applications from each other and prevent rogue applications from making the system unusable. So my suspicion is some kind of hardware fault is to blame. Is there any chance you can simply run the same test on a different PC?
If you don't have profilers, you may have to do the same work by hand...
Have you tried commenting out all read/write operations, just to check whether the slow down disappears ?
"Divide and conquer" strategies will help you find where the problem lies.
If you run it under an IDE, run it until it gets real slow, then hit the "pause" button. You will catch it in the act of doing whatever takes so much time.
You use tools like "IBM Rational Quantify" or "Intel VTune" to detect performance issue.
[EDIT]
Like Benoît did, one good mean is measuring tasks time to identify which is eating cpu.
But remember, as you are working with many files, is likely to be missing that causes the memory to disk swap.
when task manager is taking 2 minutes to come up, are you getting a lot of disk activity? or is it cpu-bound?
I would try process explorer from sysinternals. When your system is in the slowed-down state, and you try running, say, notepad, pay attention to page fault deltas.
Windows is very greedy about caching file data. I would try removing file I/O as someone suggested, and also making sure you close the file mapping as soon as you are done with a file.
I/O is probably causing your slowdown,especially if your files are on the same disk as the OS. Another way to test that would be to move your files to another disk and see if that alleviates the problem.