Our web is running on AWS with Ubuntu OS. We developed it on top of playframework. Right after the web is deployed, it is pretty quick. However, after 1 days or os, it slows down significantly. I checked resource usage of the OS, it seems normal and is responsive. Just the web service is slow to request. I suspect there are some memory, thread pool or some resource leak. Any suggestion about how to investigate it? I used 'top' and 'ps' command to look at current resource usage but they all seem normal.
You may want to create a core dump and then take that to you dev computer and examine it. This is not the easiest way but if you have limited access to the box this may be required.
Create a core dump
Analyze Core Dump File?
Related
I have a console application that runs for less than 500ms, but that according to BenchmarkDotNet allocates more than 100 MB.
I am trying to figure out what are those 100Mb because it does not add up. However I cannot find a tool to do so in Linux or Mac. Once the method the app calls is over, the GC can clean all that memory without problems, so it is not a leak I can see in a dump, unless I take the dump in the very exact moment before exiting the method. I am not clear which is the moment in which the algorithm peaks in memory usage.
I can take CPU traces using dotnet-trace and show it in the browser with Speedscope, but I cannot show in Speedscope a trace when using gc-verbose or gc-collect as provider.
Is there a way with dotnet-trace to print in the console the stats of the created objects or anything like that?
try out dotnet dump and checkout this article by Tess Ferrandez.
and maybe you can share a little bit more information please.
I'm developing applications (services) for Windows and sometimes have problem with performance and recources (especially with MsSql). I need to know which service, application or OS component, developed by my or someone else, makes load CPU or HDD at some moment in past.
I whant to be able to do it using some kind of stored data (log), better with grafics.
Is there any way to do it?
Perfmon will be you built in friend!
you can either log current performance counters in a user session or let a background service track your preselected counters and you can check that afterwards.
you will find tons of explanations how to user perfmon. It is part of every windows since NT4.
Is it possible to leave some applications running on RAM after closing them on Windows. What Im askig is like cached RAM but more like application specific, to specifically run those applications faster.
Windows does that automatically with its SuperFetch subsystem. It monitors which applications are used frequently and at what time of day and makes sure to have them cached at the right time.
And generally, when closing an application its files should still be cached, so a subsequent startup should be fast.
I am running redis on windows and I am having some performance issues. The machine is a Xeon E5 with 32GM RAM and SSD with HW-Raid with Windows Server 2012. There are some other processes running, but they are not critical and are idle most of the time.
I noticed performance problems and operations timeout very often, so I started "redis-cli --intrinsic-latency 100". The output shows that the max-latency goes up to 15000 microseconds, which is very slow I think.
I was also running a memory-profiler: The r/w-performance is not so good (5GB/sec) but I think this should not be the bottleneck. At the moment I have absolutly no idea what to try.
Can you give me some tipps how to find the performance problem?
There is no "fork" as in Linux in Windows. So when you dump your redis db, it can just "stop the world" in order to write on the disk "dump.rdb". Well, they did implement a "Copy-on-write" strategy that don't stop redis, it just copies values when dumping (the redis clients will still be able to get responses from redis). It is in their version log: https://github.com/MSOpenTech/Redis
There is a replacement for the UNIX fork() API that simulates the copy-on-write behavior using a memory mapped file.
This is the real bottleneck of redis in windows as it is an overhead and is more complex (bugs?). It is explained here:http://blogs.msdn.com/b/interoperability/archive/2012/04/26/here-s-to-the-first-release-from-ms-open-tech-redis-on-windows.aspx
As a result you could try running a redis on Linux to test if this is a performance issue of the windows port. Also, the more you write a dump.rdb, the bigger is the overhead (you can change the frequency or try disabling it completely for testing).
Finally, it could also be a network problem and you should check if it is not a network rule / hardware problem (not enough throughput! Bad cable or stuff, firewalls...). Are your redis clients on the same hardware machine?
I have been using a Windows port of Redis called "Memurai". They have a developer edition free of charge.
Now, in one of their blog they claim they have solved the fork() problem. See excerpt below.
Memurai performance seems good to me, even with persistence enabled (both RDB and AOF) although I have not run any specific test myself. There's another blog about Memurai perf in here.
It's worth giving it a try.
"Internally, Redis uses the fork() system call to perform asynchronous writes, but that’s not an option for Memurai because fork() doesn’t exist on Windows. Instead, Memurai uses Windows shared memory to implement a start-of-the-art version of fork() that’s finely tuned for performance and..."
Our ASP.Net 2.0 web app was running happily along on Windows Server 2003. We were starting to see some of the limits of the environment approaching, such as memory and CPU usage spikes, and as we're getting ready to scale we decided it was time for a larger server with higher availability.
We decided to move to Windows Server 2008 to take advantage of IIS 7's shared configuration. In our development and integration environments, we reproduced the OS and app in 2008/IIS 7 and everything seemed fine. But truth be told, don't have a good way of simulating production-like loads as of yet, nor can we reproduce our prod environment accurately (we're small with limited resources). So once we rolled out to production, we were surprised to find performance significantly worse on 2008 than it was on 2003.
We've also moved from a 32-bit environment to 64-bit in the process, and we've also incorporated ASP.Net 3.5 dll's into the project.
Memory usage is through the roof, but I'm not as worried about that. We believe in part this is because of the overhead with Server 2008's memory, so throwing more RAM at it may solve that issue. The troubling thing is we're seeing processor spikes to 99% CPU Utilization, which we've never seen before in the 2003/IIS 6 environment.
Has anyone encountered these issues before and are there any suggestions for a solution/places to look? Right now we're doing the following:
1) Buying time by adding memory.
2) Buying time by setting app pool limits: shut down w3wp.exe when CPU hits 99% load. Since you don't have the option to recycle the app pools, I have a scheduled task running that recycles any stopped app pools.
3) Profiling the app pools under Classic and Integrated modes to see which may perform better.
Any other ideas are completely welcome.
Our experiance is that code runs much faster on a 64bit windows 2008 than on a 32bit windows 2003 server.
I am wondering if something else is also running on the machine. For example is SQL Server installed with a maintainence plan that could cause the CPU spike.
I would check the following:
Which process is using the CPU?
Is there a change in the code? Try installing the new code on the old machine
Is it something to do with the compile options? Is the CPU usage a recompile?
Are there any errors in the event log?
In our cases, since we have 4 processors, we then increased the "number of worker process to 4" currently working well so far as compare before.
here a snapshot:
http://pic.gd/c3661a
You can use the application pool "Recycle" option in IIS7+ to configure physical and virtual memory limits for application pools. Once these are reached the process will recycle and the resources will be released. Unfortunately the option to recycle based on CUP usage has been removed from IIS7+ (some one correct me if I'm wrong). If you have other apps on the server and want to avoid them competing for resources when this condition happens you can implement Windows System Resources Manager and it's IIS policy (here is a good tutorial http://learn.iis.net/page.aspx/449/using-wsrm-to-manage-iis-70-apppool-cpu-utilization/)
Note SRWM is only available on Enterprise and Data Center editions.