I am running UI tests using NightwatchJS v 1.1.11 and Selenoid
selenoid is hosted on a t2.micro EC2 instance(evaluation phase hence using t2.micro).
Tests randomly fail in the middle of tests for 2 main reasons:
1) socket hang up exceptions
13:10:05 tests/readOnlyProduction/palette.js POST http://xx.xx.xx.xx:4444 /wd/hub/session/87f6eb1b0b3369ba93bb9ce8c548fa89/element/0.46204216593315195-1/click - ECONNRESET
13:10:05 tests/readOnlyProduction/palette.js Error: socket hang up
13:10:05 tests/readOnlyProduction/palette.js at createHangUpError (_http_client.js:342:15)
13:10:05 tests/readOnlyProduction/palette.js at Socket.socketCloseListener (_http_client.js:377:23)
13:10:05 tests/readOnlyProduction/palette.js Error while running .clickElement() protocol action: An unknown error has occurred.
2) session timed out or not found
13:11:06 tests/readOnlyProduction/palette.js Error while running .locateMultipleElements() protocol action: Session timed out or not found
Tests are being triggered from Jenkins running against selenoid in EC2 but similar exceptions are observed when selenoid is hosted in local and tests are triggered in local.
Most importantly, this happens when tests are run in parallel while same tests works fine when run one at a time.
So far tried following, but no luck so far.
1) Adding this in nightwatch.json
"request_timeout_options": {
"timeout": 100000,
"retry_attempts": 3
}
2) I had lots of test cases inside single test suite, so broke into smaller tests.
Several things to check:
1) Check that your VM has sufficient computing resources (CPU, memory, disk). This can be usually verified using AWS CloudWatch (more on this). Our recommendation is to start with ~1.5-2 browsers per CPU core and 1 Gb RAM for every browser running in parallel.
2) Try to increase Selenoid session timeout (default is 1 minute). This could be e.g. easily done with capability:
sessionTimeout: "2m"
3) Check that you don't have any network connectivity issues, e.g. static page resources like JS and CSS blocked by firewall rules.
Related
While running Jmeter in non-gui mode for 100 users for the first time after the restart of the server, it shows an "Internal Error" message for a single save thread (other 99 save threads have passed).
This error is when run for the first time.Later it passes 100 users any N number of times.
But when the same is done in manual testing (ie; saving 100 users) there is no error.
What could be the issue.
According to RFC 7231
6.6.1. 500 Internal Server Error
The 500 (Internal Server Error) status code indicates that the server
encountered an unexpected condition that prevented it from fulfilling
the request.
In general all HTTP status codes 5xx indicate a server error, so there is nothing to do with your JMeter script, you need to look into your server logs to identify the root cause of the issue on the server side.
It might be the case that the server gets overloaded and cannot handle 100 concurrent users due to one of the following reasons:
incorrect configuration of the server infrastructure (application server, database configuration settings, etc.), make sure that all the components of your application are properly configured for high load
lack of resources (CPU, RAM, Network and/or Disk IO, swap, sockets, etc.), make sure to monitor essential resources usage during the test execution, it can be done using i.e. JMeter PerfMon Plugin
it might also be the case your application uses inefficient or too complex algorithms, it worth running a test with performance profiling tools telemetry enabled, this is probably the best way of identifying the bottleneck
I am doing some benchmark testing. During this, I have to increase numbers of users 3000, keeping ramp-up time 100 and loop count 1.
Somehow jmeter giving the below errors:
Response code: Non HTTP response code: org.apache.http.conn.HttpHostConnectException
Response message: Non HTTP response message: Connection to https://the-homepage-I-am-testing.net refused
There is no such thing as "JMeter5.5" yet, the latest version as of now is JMeter 5.1
Looking into error details it looks like a bottleneck on the application under test side so you need to inspect what's causing these connections refusals:
inspect your application logs for any suspicious entries, it might be a matter of a thread pool maximum setting which is not sufficient
inspect your application server / database configuration as it might be the case you have the above limitation on the middleware level
inspect OS configuration for the same as it might be the case there is not enough maximum open handles so OS cannot allocate a socket to serve the connection
make sure to monitor baseline health metrics, application under test should have enough headroom to operate in terms of CPU, RAM, etc. - you can use JMeter PerfMon Plugin for this
Looking into your configuration it doesn't necessarily assume 3000 concurrent users as you have only 1 iteration. Given 100 seconds ramp-up it might be the case that some users have already finished test execution and some had not been yet started. Double check you're really deliver the anticipated load using i.e. Active Threads Over Time listener.
Scenario taking into account login user->Navigate to Page 01->hold the user for 5min->Logout user
Scripted as below:
Navigate to the Home page
The user is logged in (Assertion for login verification over some text on the dashboard)
Dashboard appears
Navigate to Page 01 (Assertion Page 01 content)
Logout (Constant Timer added for 5min and Assertion for logout to verify home page is redirected)
Step Up thread configuration has been kept this way:
For achieving this scenario distributed system was implemented as follows:
Master(My own machine 8 GB Ram and Core 2 Duo Processor)
2 slaves machines (8 GB Ram each and I7 and Core 2 Duo Processors)
Thread: jp#gc - Stepping Thread Group
The server has been configured as below:
2 EC2 Instance (16 GB Ram each)
1 Load Balancer
1 RDS Instance
Note: Instance is auto scaled at 60% CPU Utilization.
While executing the script for 500 concurrent users using stepping thread on Non-GUI mode, below list of error is appearing on the dashboard report
504/Gateway Time-out
Non HTTP response code: java.net.SocketException/Non HTTP response message: Connection reset
Assertion as logout failed
Could someone help me out to know why these are appearing? when I checked the Load Balancer 504/Gateway Time-out was not appearing there? I was trying to track these error but was not able to figure it out why these along with other two errors are appearing. When the same script is executed for 10 users not error is appearing on GUI mode.
While the same script when executed for 100-250 concurrent user it works pretty well when no such above error.
If the issue doesn't happen for 250 virtual users and happens for 500 - it's definitely the bottleneck caused by increased load, you just need to find out the reason.
Make sure to have DNS Cache Manager added to your Test Plan otherwise you may run into the situation when the load goes to one server only
Set up monitoring of your EC2 instances to ensure that they have enough headroom to operate in terms of CPU, RAM, Network, etc. You can use Amazon CloudWatch or JMeter PerfMon Plugin for this.
You might want to re-run the test with profiling tool telemetry enabled - this way you will be able to see where application spends the majority of time
Inspect the configuration of your application servers, databases, etc. as it might be configuration issue of the middleware
Be aware that according to JMeter Best Practices you should always be using the latest JMeter version so consider migrating to JMeter 5.0 (or whatever is the latest version available at JMeter Downloads page) on as soon as possible.
Ubuntu: Execution was not ending properly before test completion
I have two questions
1) I am running 4500 users test from two machines (2250+2250). Duration of the test is 1 hr 30 min.
At the end test was not closing and it's not giving a consolidated summary in the console.
2) after closing the test by keys(ctrl +C). If I open the jtl report from the master system its showing 2250 users in the (Active threads over time) Lisner.
Using ctrl+c is not enough for terminating the running process.
End your test by executing the shutdown.sh (or bat depending on your environment) to signal the running process to properly and gracefully end its execution. See here: http://jmeter.apache.org/usermanual/get-started.html
Edit: you can use the stoptest.cmd|sh for forcing a stop.
Looking into your execution log file my expectation is that your are kicking off 9000 threads. When it comes to remote testing JMeter Remote Engines are absolutely independent so each engine is executing the Test Plan it receives from the master.
If you define 4500 threads in the Thread Group(s) it means that every JMeter slave will kick off 4500 virtual users.
The fact that JMeter doesn't respond to Shutdown event indicates the fact that JMeter engines are overloaded so they even cannot gracefully shutdown in a reasonable time. Make sure you have monitoring of at least baseline CPU and Memory stats of JMeter load generators to ensure that they have enough headroom to operate as if they will be lacking resources it will result in the increased response time (and impossibility to shut the test down as well).
So my recommendations are:
Make sure the load you deliver matches your expectations as I am under the impression you are delivering twice as many users
Make sure you have monitoring of CPU, RAM, Network and Disk capacity on JMeter load generators, you can use i.e. JMeter PerfMon Plugin for this.
Make sure you're following JMeter Best Practices and recommendations from the 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
I'm running Jmeter load test from command prompt for about one hour and test is not ending at one hour. It is taking more time that one hour.
I'm getting errors in Jmeter logs for failed transactions:
Non HTTP response code: java.net.SocketException
Non HTTP response message: Unexpected end of file from server
Please advise. Is it because of some threads is hanging and how to rectify it.
Thank you
Most probably you didn't set Connect and Read Timeout in Http Request default.
So if you server hangs, JMeter will wait infinitely for the response.
Beside ensure you set correct values in your scheduler.
If issue persists, please give more details with screenshot.
By default JMeter "asks" threads to stop and waits for their graceful shutdown. If threads cannot be stopped in a timely fashion you have the following options:
Implement a "hard stop", i.e. instead of "asking" threads to stop you need to "tell" them to stop. It can be done using i.e. Test Action sampler with Stop now action. (remember that in this case JMeter won't wait for graceful shutdown and you will get extra errors connected with abnormal threads termination)
The fact that JMeter threads are stuck may indicate problem with your application so I would recommend checking baseline health metrics on the application under test side using i.e. JMeter PerfMon Plugin. If there is enough headroom but it still works slowly or doesn't work at all it might indicate a bottleneck in your application code or infrastructure.
Also make sure you are following JMeter Best Practices and make sure your JMeter load generator(s) have enough CPU, RAM, etc. as in some cases JMeter can report "false negative" results due to lack of hardware resources.