I am transferring a C# asp.net asmx web service to windows 2008 and iis7 (64 bit), from ii6 (32 bit). I got an approximate web method performance time of about 160 ms before. On iis7, I'm now getting about 320 ms, even after reducing the web method down to almost no code to execute. I realize there is a compilation time on the first call. This timing is after about 20 calls and the time seems has stabilized.
I would like to reduce the time to run the web method from 320 ms to under 200 ms. This is to help handle the case where several calls would need to be processed. Another problem is when I ramp up 20 calls in 1 second, once in a while one of the calls will take about 3 seconds. This is also not desirable.
I've tried compiling in release mode and removed a debug compilation from the web.config. The .asmx file just references the class to load in the dll binary.
Something that is different is that iis7 is configured to show more detailed error messages to help with setup. However, since this is only when an error occurs I don't see how it could be slowing a regular call down.
I've tried both integrated pipeline mode and the classic pipeline mode and still get similar times. I've also tried setting the default compilation language to C#. I've tried checking the ping time to verify it is not the network. IIS has some database connections setup from the time when there was code in the web service method, but now that it does basically nothing I don't believe that should be an issue.
fyi - The problem was not asmx, iis7, iis6, debug mode, or the pipeline mode. The new IIS7 64 bit server started off with an internal ip address and then graduated to a public ip address. The ip address made a difference due to differences in network routing. The IIS6 test case was based on a public ip. Once I used the public ip for the IIS7 server the times were reasonably similar.
Related
We are working on an ASP.NET 5 Web API project that is in production now but we are experiencing an issue where it becomes unresponsive intermittently throughout the day.
A few notes about the application architecture. It is an ASP.NET Web API project using a MariaDB database on a separate EC2 instance within the same private network. The connection string uses the private IP of the database server to avoid any name resolution issues. The site is hosted via IIS 10.
The application itself has been developed carefully following the best practices provided by Microsoft. Heavy focus on async operations, minimizing query response times and offloading more expensive operations into background services.
The app is extremely responsive. It performs with sub 100ms responses on almost all requests, even the more complicated requests, and all the way up until it becomes unresponsive this high level of performance remains the same. We tend to see between 10-30 requests per second and 300-500 select queries per second at peak usage so not too extreme. However, randomly (2-3 times over a 24 hour period) it will begin hanging on requests and simply not respond to the request. During this time, the database is still extremely responsive and we are never over 300 connections out of our 512 connection limit.
The resources on the application server itself are never really taxed much at all. The CPU never gets above ~20% and the memory usage sits around 20-30%.
If I were to stop the site in IIS and start it again while this is happening, it will quickly come back online. If I don't it will be down for a few minutes until IIS finally kills it due to a failed health check. There are no real errors generated as a response to the issue other than typical errors caused by the hanging of the process such as connection terminated errors. The only thing I have seen before that gave me pause was the fact that there a few connection timeouts when getting the connection from the pool, but like I said, the connections to the server are never close to the limit.
Also, this app and version has been in production for months and it wasn't until the traffic volume started to grow that we started seeing these issues. At this point, I am at a loss for next steps of troubleshooting and I'm seeking suggestions.
In IIS App Pool advanced settings set Start Mode to AlwaysRunning
I never found a root cause for this issue, however, after updating to newer versions of .NET MVC this issue went away. My best guess is that changes with the Kestrel possibly resolved this issue, although, I have no idea what specific change that might have been. I have gone through the change logs a few times and didn't see anything that specifically jumped out at me.
I have a selfhost signalr application, everything is ok but when users become more than 5000, users reconnected rapidly. I know that defalt value of appConcurrentRequestLimit is 5000. and i run this:
cd %windir%\system32\inetsrv
appcmd.exe set config /section:system.webserver/serverRuntime /appConcurrentRequestLimit:100000
but nothing changed. I increased maxConcurrentRequestsPerCPU and requestQueueLimit according to this
but i have got problem yet.
i'm using windows server 2012 and iis 8
You are shooting in the dark here, and you have no data about the actual performance and what's happening. The users could reconnect because of different reasons (server timeouts, regular interval reconnects, server errors). There are countless possibilities.
The correct way to know what's happening and measure performance is to run a Baseline performance load test using the default configuration, and collect the relevant performance counters like current requests, queued requests, current connections, max connections etc.
You should also collect any relevant Error logs on the server that could help you figure out what's happening.
You can find the full list of performance counters you need below:
Memory
.NET CLR Memory# bytes in all Heaps (for w3wp)
ASP.NET
ASP.NET\Requests Current
ASP.NET\Queued
ASP.NET\Rejected
CPU
Processor Information\Processor Time
TCP/IP
TCPv6\Connections Established
TCPv4\Connections Established
Web Service
Web Service\Current Connections
Web Service\Maximum Connections
Threading
.NET CLR LocksAndThreads\ # of current logical Threads
.NET CLR LocksAndThreads\ # of current physical Threads
Once you have your baseline performance results on a graph, then you can modify configuration (e.g. modify the number of concurrent requests like you tried above) and then re-run your test, and collect again the same performance counters.
The performance counter results will speak for themselves, and they will lead you to a solution.
You can generate the load with a tool like Crank:
https://github.com/SignalR/SignalR/tree/dev/src/Microsoft.AspNet.SignalR.Crank
In addition you can also check the SignalR troubleshooting guide:
http://www.asp.net/signalr/overview/testing-and-debugging/troubleshooting
I have an asp.net mvc4 web api interface that gets about 54k requests a day.
http://myserv.x.com/api/123/getstuff?whatstuff=thisstuff
I have 3 web servers behind a load balancer that are setup to handle the http requests.
On average response times are ~300ms. However, lately something has gone awry (or maybe it has always been there) as there is sporadic behavior of response times coming back in 10-20sec. This would be for the same request hitting the same server directly instead of through the load balancer.
GIVEN:
- System has been passed down to me so there may be gaps with IIS confiuration, etc,.
- Database: SQL Server 2008R2
- Web Servers: Windows Server 2008R2 Enterprise SP1
- IIS 7.5
- Using MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs
- Looked at the logs but really don't see anything significantly relevant
- One application pool...no other LOB applications running on this server
Assumptions & Ask:
Somehow I'm thinking that something is recycling the application pool or IIS worker threads are shutting down and restarting thus causing each new request to warmup and recache itself. It's so sporadic that it's tough to trouble shoot right now. The same request to the same server comes back fast as expected (back to back N requests) since it was cached in about 300ms....but wait about 5-10-20min and that same request to the same server takes 16seconds.
I have limited tracing to go by as these are prod systems so I can only expose so much logging details. Any help and information attacking this or similar behavior somebody else has run into is appreciated. Thx
UPDATE:
The w3wpe.exe process grows to ~3G. Somehow it gets wiped out and the PID changes so itself or something is killing it every 3-4min I see tons of warnings in my webserver (IIS) log:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
After 4-5 days of assessing IIS and configuration vs internal code issues I finally found the issue with little to no help with windbg or debugdiag IIS tools. Those tools contain so much information even with mini dumps or log trace stacks that they can be red herrings. Best bet was to reproduce it by setting up a "copy intelligently" instance of a production system, which we did not have at the time and took a bit for ops to set something up.
Needless to say the problem had to do with over cacheing business objects. There was one race condition where updates on a certain table were updating an attribute to that corresponding business object (updates were coming from multiple servers) which was causing an OOC stackoverflow that pretty much caused the cacheing to recursively cache itself to death thus causing the w3wp.exe process to die and psuedo-recycle itself. It was one of those edge cases that was incredibly hard to test and repro in a non-production environment.
I have developed 2 soap webservices in my zend application. In my development environment (MAMP on mac 8 GB ram i7 processor) the performance is really good. When I deploy it on my Ubuntu LAMP server (1 GB RAM 1 processor) the performance decreases a lot. Its more than 10 times slower.
I have a java client (eclipse autogenerated client from wsdl) The problem is that the first call is always 4 times slower than the second one. This goes for both my MAMP and LAMP.
MAMP
- First call 400 ms
- Second call 100 ms
LAMP
- First call 2 000 ms
- Second call 400 ms
I simply duplicate the request so the request is exactley the same for the first and second call.
If I manually run the LAMP client several times the first call will be done at around 900 ms. It feels as if the Zend application has to "startup" something during the first call.
Does anyone have any clue on how I can get around this? What I've tried:
Make sure the wsdl is cached
Installed xcache (not shipped with LAMP)
Read tunings tutroials
Thanks in advance!
This performance issue often occurs when you use Zend_Soap_AutoDiscovery for wsdl generation. If that is the case for your code, you should consider storing your generated wsdl as a separate xml file and load it in the Zend_Soap_Server constructor.
This looks like a problem with opcode cache. Without opcode cache, Zend's really slow. And it gets a ncie boost when using it.
I'd look for Zend Optimizer, eAccelerator, or simillar...
That would be why it slows down after some idle time (classes/files are wiped from IO cache).
I am doing performance tweaking of a simple app that uses MVC on IIS 7.5.
I have a StopWatch starting up in Application_BeginRequest and I take a snapshot at Controller.OnActionExecuting.
So I measure the time spend in the entire IIS pipeline: from request receipt to the moment execution finally gets to my controller.
I obtain 700 microseconds on my 3GHz quad-core (project compiled Release x64), and I wonder where the bottleneck is, especially hearing some people say that one can get up to 8000 page loads per second with MVC.
How can I optimize MVC and IIS pipeline to obtain higher speed?
The tool "IIS Tuner" may be helpful. It is an open source tool and you may investigate the tricks that application made. the tool is available at codeplex
I obtain 700 microseconds on my 3GHz quad-core (project compiled Release x64), and I wonder where the bottleneck is, especially hearing some people say that one can get up to 8000 page loads per second with MVC.
Note that a result of 700 µs in the pipeline is not incompatible with getting throughput of 8,000 requests per second. (You may be confusing response time with throughput.) If 8,000 people simultaneously made requests and each one was fulfilled less than one second later, that would be 8,000 requests per second regardless of whether the response time was 1 µs, 700 µs, or 700 ms.
Is 700 microseconds too long for IIS+MVC pipeline to run on every page load?
Not necessarily. You'd have to evaluate whether or not you're actually getting saturated with requests.
The tool "IIS Tuner" may be helpful.
And makes WCAT not works. Any views?
more details as below,
D:\Program Files\IIS Resources\WCAT Client>wcclient.exe localhost
wcclient.exe 5.2.3652 - Web Capacity Analysis Toolkit Client.
Copyright (c) 1995-2002 Microsoft Corporation. All rights reserved.
Compiled May 29 2003, 16:28:20
Connecting main client thread...
Connected.
Waiting for Config Message: Connecting Dead controller thread...
Done.
IP version requested for testing is unspecified
Receiving script header message: Done.
Receiving string table: Receiving 1 script pages ...
Fail to resolve server address for IP supported by the client: localhost
Connecting client abort notification...
Failed to resolve server address(es).
Have you looked into:
- async controllers? ASP.NET processes are limited to 12 threads (or 12 threads per CPU) not sure which.
- there are a bunch of micro-optmization tricks (for example, MVC loads all the view engines...when you only need Razor remove the others)
So, there are definitely ways to improve performance and you have full control over the HTML in MVC as well (no viewstate, obstrusive markup, unecessary postbacks etc)