I have ASP.NET MVC app which accept file uploads and has result pooling using SignalR. The app hosted on Prod server with IIS7, 4 Gb Ram and two cores CPU.
The app on Dev server works perfectly but when I host it on Prod server with about 50 000 users per day the app become unrresponsible after five minutes of running. The web page request time increase dramatically and it takes about 30 seconds to load one page. I have tried to record all MvcApplication.Application_BeginRequest event call and got 9000 hits in 5 minutes. Not sure is this acceptable number of hits or not for app like this.
I have used ANTS Performance Profiler(not useful in Prod app profiling, slow and eats all memory) to profile code but profiler do not show any time delay issues in my code/MSSQL queries.
Also I have tried to monitored CPU and RAM spike problems but I didn't find any. CPU percentage sometimes goes to 15% but never up and memory usage is normal.
I suspect that there is something wrong with request or threads limits in ASP.NET/IIS7 but don't know how to profile it.
Could someone suggest any profiling solutions which could help in this situation? Tried to hunt the problem for two week already without any result :(
You may try using the MiniProfiler and more specifically the MiniProfiler.MVC3 NuGet package which is specifically created for ASP.NET MVC applications. It will show you all kind of useful information such as the time spend for different methods in the execution of the request.
Related
We are working on an ASP.NET 5 Web API project that is in production now but we are experiencing an issue where it becomes unresponsive intermittently throughout the day.
A few notes about the application architecture. It is an ASP.NET Web API project using a MariaDB database on a separate EC2 instance within the same private network. The connection string uses the private IP of the database server to avoid any name resolution issues. The site is hosted via IIS 10.
The application itself has been developed carefully following the best practices provided by Microsoft. Heavy focus on async operations, minimizing query response times and offloading more expensive operations into background services.
The app is extremely responsive. It performs with sub 100ms responses on almost all requests, even the more complicated requests, and all the way up until it becomes unresponsive this high level of performance remains the same. We tend to see between 10-30 requests per second and 300-500 select queries per second at peak usage so not too extreme. However, randomly (2-3 times over a 24 hour period) it will begin hanging on requests and simply not respond to the request. During this time, the database is still extremely responsive and we are never over 300 connections out of our 512 connection limit.
The resources on the application server itself are never really taxed much at all. The CPU never gets above ~20% and the memory usage sits around 20-30%.
If I were to stop the site in IIS and start it again while this is happening, it will quickly come back online. If I don't it will be down for a few minutes until IIS finally kills it due to a failed health check. There are no real errors generated as a response to the issue other than typical errors caused by the hanging of the process such as connection terminated errors. The only thing I have seen before that gave me pause was the fact that there a few connection timeouts when getting the connection from the pool, but like I said, the connections to the server are never close to the limit.
Also, this app and version has been in production for months and it wasn't until the traffic volume started to grow that we started seeing these issues. At this point, I am at a loss for next steps of troubleshooting and I'm seeking suggestions.
In IIS App Pool advanced settings set Start Mode to AlwaysRunning
I never found a root cause for this issue, however, after updating to newer versions of .NET MVC this issue went away. My best guess is that changes with the Kestrel possibly resolved this issue, although, I have no idea what specific change that might have been. I have gone through the change logs a few times and didn't see anything that specifically jumped out at me.
I'm using Sitecore instance 9.1, Solr 7.2.1, and SXA 1.8.
I have deployed the environment on Azure and while monitoring incoming requests (to CD instance), I've noticed slowness in loading some pages at specific times.
I've explored App Insights and found an unexplainable behavior the request is taking 28.7 seconds while the breakdown of it shows executions of milli-seconds .. How is that possible? and How to explain what's happening during extra 28 seconds on the app service ??
I've checked the profiler and it shows that the thread is taking only 1042.48 ms .. How is that possible ?
This is an intermittent issue happens during the day .. regular requests are being served within 3 to 4 seconds.
I noticed that Azure often shows a profile trace for a "similar", but completely different request when clicking from the End-to-end transaction view. You can check this by comparing the timestamp and URL of the profile trace and the transaction you clicked from.
For example, I see a transaction logged at 8:58:39 PM, 2021-09-25 with 9.1 s response time:
However, when I click the profile trace icon, Azure takes me to a trace that was captured 10 minutes earlier, at 08:49:20 PM, 2021-09-25 and took only 121.64 ms:
So, if the issue you experience is intermittent and you cannot replicate it easily, try looking at the profile traces with the Slowest wall clock time by going to Application Insights → Performance → Drill into profile traces:
This will show you the worst-performing requests captured by the profiler at the top of the list:
In order to figure out why it is slow, you’ll need to understand what happens internally, f.e:
How the wall clock time is spent while processing your request?
Are there any locks internally?
The source of that data is dynamic profiling, Azure can do that on demand.
The IIS stats report would show you slowest requests, so you could look into Thread Time distribution to see where those 28 seconds are spent:
In Sitecore the when the application start the Initial prefetch configuration allows to pre-populate prefetch caches. Pre-heated prefetch caches help to reduce the processing time of incoming requests. The initial prefetch configuration of caches are taking time to load on initial stage.
Sitecore XP instance takes too long to load. This is caused by a performance issue in the CatalogRepository.GetCatalogItems method. It will be fixed in upcoming updates
see Site core knowledge base
In Sitecore XP 9.0 the initial prefetch configuration was revised. The prefetch cache for the core database was configured to include items that are used to render the Sitecore Client interface.
The Sitecore Client interface is not used on Content Delivery instances. Disabling initial prefetch configuration for the Core database helps in avoiding excessive resource consumption on the SQL Server hosting the Core database.
Change the configuration of the Core database in the \App_Config\Sitecore.config file:
Refer site core knowledge base
I'm thinking about launching an web app with heroku but I have no idea to calculate the performance cost. According to their website 1 dyno to professional support is at least $ 25 /month gives a machine with "512MB or 1GB RAM".
If my website has a standard load assets and I have 1,000 people everyday access, how many "dynos" should be recommended to have a reasonable good speed for users?
This will likely get shut down as it's not a 'programming' question. I'll try to answer quickly.
It all depends on how fast your page executes on the server, if it's slow then you run the risk of requests being timed out before they can be served by the app server. Adding more dyno's does not make your application faster though, changing the dyno type, standard-1x, 2x, PM, PL does as it makes more resources available however, slow code will always be slow code - throwing more resources at it is like a band-aid for a period of time.
In short, it's very application dependent both from a code point of view and traffic point of view - do those 1000 users arrive at the same time or are they spreadout?
I have an MVC3 application which is working fast in my dev environment (even when pointed at the production database). However, when I publish the application and move it onto the production iis7 environment it runs at a snails pace. I understand that the inital load can take a few seconds as the application pool starts up, but this is taking 20+ seconds. Then will be fast for a few clicks and the next click will again take 20+ seconds.
I've put in the MVCMiniPorifler and it doesn't look like the database is causing problems. But, I also can't see what is causing the problem. I can hit the same page multiple times and it comes back in a second or 2 and then suddenly that same page will take 20+ seconds to respond.
Has anyone seen this sort of behaviour before? Any help would be greatly appreciated and I'm not sure what to try next.
It's possible that the other web apps running on your production server are locking required resources. Is there a common file or folder that multiple sites utilize? Are you sharing the app pool between any of the sites?
I am doing performance tweaking of a simple app that uses MVC on IIS 7.5.
I have a StopWatch starting up in Application_BeginRequest and I take a snapshot at Controller.OnActionExecuting.
So I measure the time spend in the entire IIS pipeline: from request receipt to the moment execution finally gets to my controller.
I obtain 700 microseconds on my 3GHz quad-core (project compiled Release x64), and I wonder where the bottleneck is, especially hearing some people say that one can get up to 8000 page loads per second with MVC.
How can I optimize MVC and IIS pipeline to obtain higher speed?
The tool "IIS Tuner" may be helpful. It is an open source tool and you may investigate the tricks that application made. the tool is available at codeplex
I obtain 700 microseconds on my 3GHz quad-core (project compiled Release x64), and I wonder where the bottleneck is, especially hearing some people say that one can get up to 8000 page loads per second with MVC.
Note that a result of 700 µs in the pipeline is not incompatible with getting throughput of 8,000 requests per second. (You may be confusing response time with throughput.) If 8,000 people simultaneously made requests and each one was fulfilled less than one second later, that would be 8,000 requests per second regardless of whether the response time was 1 µs, 700 µs, or 700 ms.
Is 700 microseconds too long for IIS+MVC pipeline to run on every page load?
Not necessarily. You'd have to evaluate whether or not you're actually getting saturated with requests.
The tool "IIS Tuner" may be helpful.
And makes WCAT not works. Any views?
more details as below,
D:\Program Files\IIS Resources\WCAT Client>wcclient.exe localhost
wcclient.exe 5.2.3652 - Web Capacity Analysis Toolkit Client.
Copyright (c) 1995-2002 Microsoft Corporation. All rights reserved.
Compiled May 29 2003, 16:28:20
Connecting main client thread...
Connected.
Waiting for Config Message: Connecting Dead controller thread...
Done.
IP version requested for testing is unspecified
Receiving script header message: Done.
Receiving string table: Receiving 1 script pages ...
Fail to resolve server address for IP supported by the client: localhost
Connecting client abort notification...
Failed to resolve server address(es).
Have you looked into:
- async controllers? ASP.NET processes are limited to 12 threads (or 12 threads per CPU) not sure which.
- there are a bunch of micro-optmization tricks (for example, MVC loads all the view engines...when you only need Razor remove the others)
So, there are definitely ways to improve performance and you have full control over the HTML in MVC as well (no viewstate, obstrusive markup, unecessary postbacks etc)