Performance - High Context Switch - performance

I have an application which exposes a web service on which I am trying to do a load test.
It works for few concurrent users without any issue.
When I increase the user count to 30, I simply get this error in JMeter within 100 milli seconds.
Non HTTP response code: java.net.SocketException - Non HTTP response message: Connection reset
[I thought my JMeter config was wrong - but one of the web application which uses this web-service also failed consistently around that time saying the service was unavailable. So, server itself has some issue].
I checked the web service - application log - No exception & very clean.
CPU, Memory utilization of server is also very normal on the server machine.
However, 'Context Switch' & 'Device Interrupts' are increasing under load.
Context Switch is avg 1500/sec under heavy load. Normally It is 500/Sec.
Is this bad? Is it what makes my application perform badly? I have no clue to resolve this issue.
Note: It is JBOSS server

Related

Jmeter throwing "socketexception: connection reset" during the tests. But no error can be seen on server log [duplicate]

The target for us to achieve is 500 concurrent users.
We have tried running a test for 100 users over 3 machines. And it ran fine without any errors.
When i tried running the test for 150 or More users with same number of machines, i started getting the following response code
Response code:Non Http Response code:java.net.socketException
Response message:Connection Reset
I have also tried increasing the number of machines to 8 machines. Still it is of no help. Response time is also very high (156 seconds) for some of the requests.
When we checked the server logs to find out what could be causing this issue, No error logs were found there during the time of the execution.
I'm having a hard time finding out what could be the issue. The server side is ruling out if there could be an issue from their end.
Tried the following fixes from Jmeter side:
Increasing the heap size
Changing the retry count in user.properties file
Changing Boolean=True in hc.parameters file
Used HTTP Request Defaulters to change the implementation to HTTPClient4
CPU Config:
Intel (R) Xeon(R) CPU E5-2690 v3 # 2.60 GHz (2 Processors)
5 GB Ram
64-bit Operating System
The Connection Reset error means failed attempt to write to the socket which has been closed already, on TCP protocol level it means receiving a TCP RST
It might be the case JMeter is closing the connection prematurely as JMeter 5.0 had httpclient4.time_to_live property set to 2000 and if you're seeing response times > 2 seconds (and you do) most probably JMeter is closing the connection before getting the full response.
You can try increasing this setting to 60000 matches modern browsers default settings or even more to match your application response time (if you think 3 minutes is acceptable) or consider upgrading to JMeter 5.3 which has better default value.
More information: Bug 64289

Jmeter throws "socketexception: connection reset" error during execution

The target for us to achieve is 500 concurrent users.
We have tried running a test for 100 users over 3 machines. And it ran fine without any errors.
When i tried running the test for 150 or More users with same number of machines, i started getting the following response code
Response code:Non Http Response code:java.net.socketException
Response message:Connection Reset
I have also tried increasing the number of machines to 8 machines. Still it is of no help. Response time is also very high (156 seconds) for some of the requests.
When we checked the server logs to find out what could be causing this issue, No error logs were found there during the time of the execution.
I'm having a hard time finding out what could be the issue. The server side is ruling out if there could be an issue from their end.
Tried the following fixes from Jmeter side:
Increasing the heap size
Changing the retry count in user.properties file
Changing Boolean=True in hc.parameters file
Used HTTP Request Defaulters to change the implementation to HTTPClient4
CPU Config:
Intel (R) Xeon(R) CPU E5-2690 v3 # 2.60 GHz (2 Processors)
5 GB Ram
64-bit Operating System
The Connection Reset error means failed attempt to write to the socket which has been closed already, on TCP protocol level it means receiving a TCP RST
It might be the case JMeter is closing the connection prematurely as JMeter 5.0 had httpclient4.time_to_live property set to 2000 and if you're seeing response times > 2 seconds (and you do) most probably JMeter is closing the connection before getting the full response.
You can try increasing this setting to 60000 matches modern browsers default settings or even more to match your application response time (if you think 3 minutes is acceptable) or consider upgrading to JMeter 5.3 which has better default value.
More information: Bug 64289

How do I solve Jmeter Error Connection reset?

I am running a performance test on a site using Jmeter. Using a load of up to 100 simultaneous users(Threads) the tests pass perfectly, trying to raise this load to 300 users (Threads) I get the following error:
Non HTTP response code: java.net.SocketException / Non HTTP response message: Connection reset
The error occurs in only 0.68% of requests (out of 2412 requests made by 300 users(Threads) only 2 requests generated this error)
I thought it was the maximum number of connections allowed on my server, I went to my application's webconfig and entered the following information: "Min Pool Size = 5; Max Pool Size = 500;". but still not solve the problem.
Does anyone know what I can do to not generate these errors?
Most probably it indicates a problem with your application, try checking:
application logs
application/web server logs and configuration
underlying operating system logs and networking configuration. also pay attention to number of open ports/sockets/handles (can be checked using either built-in OS monitoring tools or JMeter PerfMon Plugin)
If you're absolutely sure that there is nothing wrong with your test script and application and JMeter is configured to behave exactly like a real browser you can follow instructions from JMeterSocketClosed wiki page
More information: The Mysteries of Connection Close

How to find the server processing time by IIS logs?

From IIS logs, I observed that it makes the entry in the logs once it gets acknowledgement from the client machine that requests are properly served. Hence, from the time taken the attribute of IIS logs we can say that it has the client waiting time as well.
Time Taken In IIS Logs = Server Processing Time + Client Wait Time to download the response
These are the values that we get in IIS logs:
Date (date)
Time (time)
Client IP Address (c-ip)
User Name (cs-username)
Method (cs-method)
URI Stem (cs-uri-stem)
URI Query (cs-uri-query)
Protocol Status (sc-status)
Win32 Status (sc-win32-status)
Bytes Sent (sc-bytes)
Time Taken (time-taken)
Host (cs-host)
User Agent (cs(User-Agent))
Referer (cs(Referer))
My question is that - Is there a way to find out only server processing time for each web request which serves by IIS?
Is there a way to find out only server processing time for each web
request which serves by IIS?
Probably not with regular IIS logs, but try "Advanced Logging"
See
http://www.iis.net/learn/extensions/advanced-logging-module/advanced-logging-for-iis-custom-logging#custom

Amazon ELB + jmeter

I am testing my application using jmeter tool. I am having 2 EC2 m1.small instances behind an ELB (not autoscaling group), 2 caching nodes huge RDS db (Multi-AZ). My Apache (peforked) is configured with default values like 256 maxclients. Both can handle 256-256 request. Now when jmeter throws 500 request I see connection time out error in one of sampler of the jmeter. Can anyone figure out what the problem is?
Thanks in advance.
What does JMeter throw back?
There are couple of things that could happen.
Connection reset: If this error is thrown out from your JMeter then it means that the server has maxed out and cannot support any more concurrent requests. i.e. the 256 threads allotted are all in use serving other requests and this basically means you have hit your limit on the server.
"Address already in use" exception: These kinds of errors are what you must not get when you do load tests. This basically means that there are no available ports on your system make more requests and all ports are busy. This could happen for a variety of reasons but you could try tweaking system settings like ulimit for linux or if your using a windows box you may want to look at the tcpTimedWaitDelay and corresponding settings to see the average turn around time for the ports to be handed over back into the active pool to be reused for the next connection. This condition is called tcp port exhaustion (http://www.outsystems.com/NetworkForums/ViewTopic.aspx?TopicId=6956&Topic=How-to-tune-the-TCP%2FIP-stack-for-high-volume-of-web-requests)
TO get around this you could also try distributed load testing and/or use timers to ensure that you always have the ports to make new connections.

Resources