I set up a web project with springboot and modified the Tomcat configuration：
tomcat:
max-threads: 800
accept-count: 1000
max-connections: 10000
min-spare-threads: 100
I deployed the project to a Linux server. The concurrency performance was tested with jmeter both locally and on the server. jmeter sets the following parameters：Number of Threads is 200, Ramp-up period is 0, and Loop Count is 1.
The local result is normal, and all requests are all accepted. However, 429 errors occurred in 43.5% of requests from the server test result.
How to solve this problem?
It might be the case you have a rate-limiter implemented somewhere so depending on the configuration applied you can:
Just turn it off via Spring Boot configuration
Or check your app performance within the configured maximum requests rate
Or try to impersonate different users coming from different IP addresses by setting up IP Spoofing in your JMeter test so your server would "think" that users are coming from the different locations and apply rate limiting for each user and not globally.
Related
I have created a performance test suite in Jmeter. The suite contains one thread group:
Thread properties:
Number of threads: 100
Ramp-up period: 5
Loop count: 1
Clearing cache and cookies on each iteration.
Thread group has 30 samplers (HTTP Requests for 30 pages), each samplers has "View Results Tree" and "View Results in table" listeners.
With this I am running the test suite.
Problem: I am getting "502 Response" for some requests on some threads.
I am new to performance testing and Jmeter. Please let me know what is the reason for getting 502 in this case and how can I solve this.
All HTTP status codes 5xx are server errors which means that you cannot "solve" them from JMeter perspective.
Particular HTTP status code 502 means "Bad gateway", as per description:
The HyperText Transfer Protocol (HTTP) 502 Bad Gateway server error response code indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server.
Note: A Gateway might refer to different things in networking and a 502 error is usually not something you can fix, but requires a fix by the web server or the proxies you are trying to get access through.
So in order to "solve" it you need to inspect your application and middleware logs and see what is the root cause of the problem.
If it happens only under the load of 100 users and not reproducible with less amount of threads it indicates that your application is overloaded hence cannot properly respond to certain requests. So you can just report it as a performance bottleneck.
A good idea would be implementing monitoring of the system under test to measure CPU, RAM, Network, Disk usage, etc. as the reason could be as simple as the lack of resources, it can be done using JMeter PerfMon Plugin
If the machine is not overloaded but you're getting this error it means that either system under test is not properly configured for high loads or uses inefficient algorithms somewhere, you can discover what's going on under the hood by looking at profiler tool or APM tool output
What does mean connection in jmeter ? Can we get the details of how many connections were made (not the user sample but the connections)
You can monitor the number of the TCP connections open by JMeter by invoking netstat command in the OS Process Sampler
If you're testing a web application the number of connections will vary from 1 to 6 (or whatever you specify under HTTP Request Defaults)
However normally you should not be very interested in this form of information. My expectation is that you should rather be looking for Server Hits Per Second custom chart which reports how many "hits" were generated by JMeter during the test execution time frame. The listener is available via 5 Additional Graphs bundle and can be installed using JMeter Plugins Manager
I activate HTTP/2 support on my server. Now i got the problem with AJAX/jQuery scipts like upload or Api handling.
After max_input_time of 60sec for php i got: [HTTP/2.0 504 Gateway Timeout 60034ms]
with HTTP/1 only a few connections where startet simultaneously and when one is finished a nother starts.
with HTTP/2 all starts at once.
when fore example 100 images would uploaded it takes to long for all.
I don't wish to change the max_input_time. I hope to limit the simultaneous connections in the scripts.
thank you
HTTP/2 intentionally allows multiple requests in parallel. This differs from HTTP/1.1 which only allowed one request at a time (but which browsers compensated for by opening 6 parallel connections). The downside to drastically increasing that limit is you can have more requests on the go at once, contending for bandwidth.
You’ve basically two choices to resolve this:
Change your application to throttle uploads rather than expecting the browser or the protocol to do this for you.
Limit the maximum number of concurrent streams allowed by your webserver. In Apache for example, this is controlled by the H2MaxSessionStreams Directive while in Nginx it is similarly controlled by the
http2_max_concurrent_streams config. Other streams will need to wait.
I do have architecture of a Load balancer followed by two Web Application server and Database, I am hitting thousands of HTTP requests to the server from Jmeter distributed testing environment.
At the time of getting response back, few request does not get response back from the server.
I checked Database logs, 100 % requests were responded.
Checked with Web Application servers access logs, 100 % requests were responded.
Can Load balancer cause the damage traversing these pending responses to the respective clients?
Every time different different request are getting stuck.
Thanks in Advance!!
If you suspect load balancer, look at 3 typical causes first:
Server takes longer to respond than load balancer is waiting
Client has shorter timeout than it takes for server to respond.
Port/thread/connection exhaustion on load balancer, or other LB configuration problems
In all three cases, I suggest looking at the load balancer logs. Since you didn't specify which LB you are using, I cannot say exactly how the log looks, but typically LB log gives you option to see:
How long it took for a request to be sent to a web server and for the response from the web server to return to load balancer. You can them compare those numbers to timeouts configured for load balancer and the client (problem 1 and 2).
How long it took for a request from the client to be processed by LB and how long LB took to respond to a client. If it takes long, then something is not right with load balancer (problem 3)
And then of course if you have any errors on load balancer, they may just explain what's going on.
If you cannot review logs for load balancer, I suggest changing your JMeter test temporarily to target servers behind load balancer directly. You can even configure your script to evenly distribute load between all servers (for example by using multiple thread groups). That would allow you to isolate the problem, and get more information on what's going on.
I am testing a clustered application, and want to define a different thread to test each machine in the cluster. I have a thread group with two HTTP Requests, a login that is done once, and a POST that is repeated many times. I set up a CSV Data Set Config to configure the Server Name and Port Number for the HTTP Requests. A HTTP Cookie Manager holds the login cookie resulting from the login request.
The problem is that for each time a HTTP Request runs the CSV file is read again. Often the same thread reads hostname and port values, and the login cookie is then not sent because it only applies to the original machine the thread logged into.
Is there any way to get make the CSV Data Set Config only read once per thread? Or perhaps some other way to set the hostname/port on a per thread basis instead of a per request basis like I am doing now?
I believe default settings of the CSV Data Set Config provide what you're looking for:
Other option is to distribute the threads using If Controller like:
If Controller, condition: (${__threadNum} % 2 == 0)
HTTP Request for even threads
If Controller, condition: (${__threadNum} % 2 != 0)
HTTP Request for odd threads
Both options use __threadNum() function which basically returns the number of current thread, maybe you'll be able to test your approach or come up with a better one knowing this.
In general testing clustered applications by hitting different hosts separately isn't the best idea. End user usually accesses such applications via software of hardware Load Balancer so he sees single hostname or IP address. The load balancer distributes the requests between servers using various algorithms.
I think this is the situation you need to simulate and it is better to add DNS Cache Manager to your test plan so each thread could resolve hostname on its own.
You can use HTTP request Defaults.
Update your Server IP, Port and protocol details. All the HTTP requests in Thread Group will take the default ip, port etc from the HTTP request Defaults.
Check here for more details