Scenario: we have a php 7 web application using IIS10 via FastCGI. We did performance testing on our application and noticed a slowdown. We ruled out the database tier. We looked at the IIS server and, even under heavy load, when the app slows down dramatically, the resources are not strained... no CPU spikes no RAM spikes. Digging further we came to the conclusion that all the requests coming in are simply being queued. When some of requests take longer (for some very large reports that take 1-3 minutes) then every other request is being queued up waiting for the first ones to go through.
So, the question is: where do we look to increase the number of concurrent requests IIS can handle at one time?
I have found these settings, under FastCGI, but VERY little documentation. Can someone explain what these four settings do?
Instance MaxRequests
Max Instances
Queue Length
Rapid Fails PerMinute
Are there any other settings we should be looking at under DefaultAppPool?
Queue Length
Maximum Worker Processes
Recycling
UPDATE:
A few things should be clarified for other who might search this:
a request in this context means one call from the browser to the IIS server
we used jmeter https://jmeter.apache.org/ to do some basic load testing
a request (for the sake of this context) from browser to IIS server will be processed like this: browser > DefaultAppPool worker process (this is defaulted to 1 in IIS look into web gardens if you want to increase this) > FastCGI instance (think of this as process threads, language is tricky and people on the web use threads/processes/instances interchangeably which can be confusing). FastCGI is defaulted to 4. So this means when 5 concurrent requests come through, they all get funneled through the 1 DefaultAppPool worker process and 4 of the 5 will be worked on concurrently by FastCGI and the 5th will be queued. We tested this: after an IIS restart there will be no w3wp.exe or php-cgi.exe processes running. When the 5 concurrent requests come in: w3wp.exe will get started and it will spawn 4 php-cgi.exe processes (check task manager)
to increase concurrency we set FastCGI "Max Instances" to 0 which allows IIS to decide how much it wants to handle based on resources available but you can also set it to a specific higher number. We tested this and I believe it to be accurate. You can see the w2wp.exe process and php-cgi.exe processes increasing in number as requests are coming in.
you should also be able to increase the DefaultAppPool worker processes, if you were to set this to 4 and leave the FastCGI instances to 4 as well that should mean, in theory, that each worker process would spawn its own 4 FastCGI instances for a total of 4x4=16 concurrent requests. I have not tested sufficiently to be sure 100% that that's how this actually plays out.
Instance MaxRequests: Controls the FastCGI process-recycling behavior. Specifies the maximum number of requests that a FastCGI application is allowed to handle before the process is recycled. The default value is 200.
Max Instances: Specifies the maximum number of FastCGI processes to allow in the application process pool for the selected FastCGI application. This number also represents the maximum number of concurrent requests that the FastCGI application can handle. The default value is 4.
Queue Length: Specifies the maximum number of requests that are queued for the FastCGI application pool. When the queue is full, subsequent requests return the HTTP error code 503 (Service Unavailable) to clients. This error code indicates that the application is too busy. The default value is 1000.
Rapid Fails PerMinute: Specifies the maximum time allowed for a request to the application. If a FastCGI process takes longer than the specified time on a single request, it is terminated. The default value is 90 seconds.
Application Pools:
Queue Length: Indicates to HTTP.sys how many requests to queue for an application pool before rejecting future requests. The default value is 1000.
Maximum Worker Processes: Indicates the maximum number of worker processes that would be used for the application pool.
For the attributes in Recycling, you can refer to this link:
https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/recycling/
It may also be important if you say:
Digging further we came to the conclusion that all the requests coming in are simply being queued. When some of requests take longer (for some very large reports that take 1-3 minutes) then every other request is being queued up waiting for the first ones to go through.
if this is true for one session, it may be also the session-handler of php
if you are using php sessions, and the default configuration is files, you will encounter the described problem, because the first request locks the session file, and a second request has to wait until release
you can, for example use another session-handler, i preferred wincache a long time, now i use memcached.
both are able to be used as session-save-handlers, and both are non-blocking
Related
I am running a load test using JMeter with 200 users for approx 1hr. So, the observation is that a few threads are stuck even after the duration completes. Like 60 out of 200 get stuck. When I take the thread dump and observe that these threads are in a Runnable state. Any suggestions for resolving this issue? And I do not see anything meaningful from the JMeter log file.
You will find an unexpected increase in response time at the end of that time.
This is because of the thread's insufficient ramp-down time. Some of your threads were active and made requests to the server and didn't receive the response but threads were closed forcefully. If your JMeter test is stopped forcefully, all the active threads will be closed immediately. So the requests generated by those threads will get higher response time.
You can use Ultimate Thread Group for graceful shutdown time(ramp-down time) of threads just like the ramp-up time.
Here is an example setting:
This is not a normal behaviour for a JMeter test, most probably it indicates that either JMeter engine is overloaded (not properly configured for high loads) or the machine where JMeter is running is overloaded (i.e. lacks RAM and starts intensive swapping)
Make sure to follow JMeter Best Practices (run your test in non-GUI mode, remove all Listeners and test elements you don't need, increase JMeter heap size, etc.)
Make sure to monitor the essential health metrics of the machine where JMeter is running (CPU, RAM, Network and Disk IO, Swap file usage). You can use JMeter PerfMon Plugin for this if you don't have any better software
It might be the case you'll have to switch to Distributed Testing, 200 virtual users doesn't seem to be a "high" load to me, but it depends on what exactly these users are doing, if they're uploading/downloading large files it may be sufficient to cause the problems
Going forward consider adding the thread dump and jmeter log file contents to your question as it doesn't contain any clues so we can only come up with "blind shot" answers
You may want to check your HTTP timeouts.
I usually set Connect Timeout to 5000 milliseconds, and Response Time out to 30000.
Your values may vary for your specific environment/ application.
In this way, if things go bad on the server under test, all requests terminate within the timeout (with errors).
You have also to consider that, if you are retrieving an HTML page with all its embedded objects, and the web server is stuck, you need to wait for multiple timeouts to expire before the operation terminate.
I am doing some benchmark testing. During this, I have to increase numbers of users 3000, keeping ramp-up time 100 and loop count 1.
Somehow jmeter giving the below errors:
Response code: Non HTTP response code: org.apache.http.conn.HttpHostConnectException
Response message: Non HTTP response message: Connection to https://the-homepage-I-am-testing.net refused
There is no such thing as "JMeter5.5" yet, the latest version as of now is JMeter 5.1
Looking into error details it looks like a bottleneck on the application under test side so you need to inspect what's causing these connections refusals:
inspect your application logs for any suspicious entries, it might be a matter of a thread pool maximum setting which is not sufficient
inspect your application server / database configuration as it might be the case you have the above limitation on the middleware level
inspect OS configuration for the same as it might be the case there is not enough maximum open handles so OS cannot allocate a socket to serve the connection
make sure to monitor baseline health metrics, application under test should have enough headroom to operate in terms of CPU, RAM, etc. - you can use JMeter PerfMon Plugin for this
Looking into your configuration it doesn't necessarily assume 3000 concurrent users as you have only 1 iteration. Given 100 seconds ramp-up it might be the case that some users have already finished test execution and some had not been yet started. Double check you're really deliver the anticipated load using i.e. Active Threads Over Time listener.
I am trying to simulate 100 threads request with 4 different users, all the 97 requests were sent and received but the remaining 3 requests still showing as executing.
I have to manually stop the Jmeter and then only the remaining request are passed.
Why is this happening?
Your machine probably doesn't have the resources to execute the requests or Java isn't configured appropriately. Take a look at this article and this one
In case you are running jmeter in a window mode, please check for attributes provided for heap memory etc. (those should not exceed more than half of the RAM on the machine you are running upon)
I have a .NET 3.5 BasicHttpBinding no security WCF service hosted on IIS 6.0.
I have service throttling bumped up as per MS recommendations.
My service operation is getting called a few hundreds of time conccurrently, and at some point the client gets a timeout exception (59:00, that's whats set in the server and client timeouts).
If I raise the timeout it just hits the new limit.
It seems like the application just "freezes" somewhere and we have not been able to figure out how this happens.
WCF tracing on the server side doesn't come up with anything.
Any ideas regarding what could be the issue?
Thanks
I assume your WebService is not using the new async/await especially wrt the database calls. In that case its because you are blocking your limited threads.
In more detail. IIS/ASP.net only creates a limited number of threads to handle requests. The first...say 8 requests spin up threads and start working. At some point they will hit the database (I am assuming a traditional n-tier app). Those threads sleep. The next say...992 requests hit IIS and are held in a queue.
At some point the database calls return, process stuff...send data to the client. Another 8 requests are dequeued...hit the database...etc...
However each set of 8 requests takes a finite time to complete. With over 900 requests ahead of them, the last 100 or so threads will take at the very least 100 * latency * number of roundtrips before they can start up. If your latency * number of roundtrips is high...your last request will take a long time before it even gets dequeued, hence the timeout.
Two remedies exists. The first, create more threads....will use up all your memory and your IIS crashes. The second is to use .net 4.5 and async/await.
See here for more information
I find that WCF service will take 8-10 seconds to load the first hit. After that it will take less than a second.
Any thoughts?
Probably due to .NET's cold start. Have you looked at setting up the IIS Warmup Module which initializes dependancies before an initial request?
From the Learn IIS website
Decrease the response time for first requests by pre-loading worker processes. The IIS Application Warm-Up module lets you configure the Web application to be pre-loaded before the first request arrives so that the worker process responds to the first Web request more quickly.
Increase reliability by pre-loading worker processes when overlapped recycling occurs. Because the recycled worker process in an overlapped recycling scenario only communicates its readiness and starts accepting requests after it finishes loading and initializing the resources as specified by the configuration, pre-loading the dependencies reduces the response times for the first requests.
Customize the pre-loading of applications. You can configure the IIS Application Warm-Up module to initialize Web applications by using specific Web pages and user identities. This makes it possible to create specific initialization processes that can be executed synchronously or asynchronously, depending on the initialization logic. In addition, these procedures can use specific identities to ensure a proper initialization.