I have to perform load testing on a load balanced (clustered) system composed of three servers.
Is it a good practice to test via JMeter each server ? Or maybe it would be better testing the whole cluster calling the load balancer dedicated endpoint ?
Thanks !
Well behaved load test needs to mimic real life application usage as close as possible therefore if the load balancer acts as a single entry point to the system - JMeter needs to hit this endpoint only so the whole system will be like a "black box".
With regards to distributed systems testing best practices you can also consider the following couple of areas:
Load balancers may route requests depending on the origin so it might be a good idea to implement IP Spoofing so each JMeter virtual user could have its own source IP address.
Load Balancer endpoint host(s) may have multiple IP addresses so consider DNS Cache Manager to your Test Plan so each JMeter virtual user could resolve endpoint address on its own as due to caching of DNS calls on OS or JVM level your test can hit one node only while others will be idle.
Testing the whole system is always better. I mean you can find out problem with the load balancing management. It's always better to be in the same condition as the production environment.
After your assessment of the first test. You can regulate and find out that maybe one server is slowing down the chain with another test.
The answer is both (and you should find more cases.), You need to test your system on load as much as close to real environment to know its capabillities.
But also ,for example, when upgrading a version, sometimes only a few or even one server remain online and you need to know what load it can sustain .
Related
I have an app deployed on a wildfly server on the Jelastic PaaS. This app functions normally with a few users. I'm trying to do some load tests, by using JMeter, in this case calling a REST api 300 times in 1 second.
This leads to around 60% error rate on the requests, all of them being 503 (service temporarily unavailable). I don't know what things I have to tweak in the environment to get rid of those errors. I'm pretty sure it's not my app's fault, since it is not heavy and i get the same results even trying to test the load on the Index page.
The topology of the environment is simply 1 wildfly node (with 20 cloudlets) and a Postgres database with 20 cloudlets. I had fancier topologies, but trying to narrow the problem down I cut the load balancer (NGINX) and the multiple wildfly nodes.
Requests via the shared load balancer (i.e. when your internet facing node does not have a public IP) face strict QoS limits to protect platform stability. The whole point of the shared load balancer is it's shared by many users, so you can't take 100% of its resources for yourself.
With a public IP, your traffic goes straight from the internet to your node and therefore those QoS limits are not needed or applicable.
As stated in the documentation, you need a public IP for production workloads (a load test should be considered 'production' in this context).
I don't know what things I have to tweak in the environment to get rid of those errors
we don't know either and as your question doesn't provide sufficient level of details we can come up only with generic suggestions like:
Check WildFly log for any suspicious entries. HTTP 503 is a server-side error so it should be logged along with the stacktrace which will lead you to the root cause
Check whether Wildfly instance(s) have enough headroom to operate in terms of CPU, RAM, et, it can be done using i.e. JMeter PerfMon Plugin
Check JVM and WildFly specific JMX metrics using JVisualVM or the aforementioned JMeter PerfMon Plugin
Double check Undertow subsystem configuration for any connection/request/rate limiting entries
Use a profiler tool like JProfiler or YourKit to see what are the slowest functions, largest objects, etc.
I want to create a Jmeter script to test an asp.net core application. The application acts as a load balancer that distributes request to other asp.net core app running.
I want to test the efficacy of the algorithm I used, so I am looking for how to get the number of times a user or request was sent to each of the load-balanced application.
Can I use Jmeter for this? If not what test can I do? I am using Jmeter because it will load test the application.
JMeter doesn't know anything about underlying architecture of your application so it can give you only the number of the hits you made to the load balancer via Server Hits Per Second
If you need to include the number of hits to each application which is behind the load balancer you might want to use something like JMeter PerfMon Plugin which can either read the values from the file or execute an arbitrary shell command on the remote host so if you can expose the number of hits so Server Agent could consume it and plot the charts for each backend server behind the load balancer.
It also makes sense to add DNS Cache Manager to your test plan to avoid IP resolved IP addresses caching on JVM or OS level as under certain circumstances JMeter may hit only one IP address out of several
I have a really simple setup: An azure load balancer for http(s) traffic, two application servers running windows and one database, which also contains session data.
The goal is being able to reboot or update the software on the servers, without a single request being dropped. The problem is that the health probe will do a test every 5 seconds and needs to fail 2 times in a row. This means when I kill the application server, a lot of requests during those 10 seconds will time out. How can I avoid this?
I have already tried running the health probe on a different port, then denying all traffic to the different port, using windows firewall. Load balancer will think the application is down on that node, and therefore no longer send new traffic to that specific node. However... Azure LB does hash-based load balancing. So the traffic which was already going to the now killed node, will keep going there for a few seconds!
First of all, could you give us additional details: is your database load balanced as well ? Are you performing read and write on this database or only read ?
For your information, you have the possibility to change Azure Load Balancer distribution mode, please refer to this article for details: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-distribution-mode
I would suggest you to disable the server you are updating at load balancer level. Wait a couple of minutes (depending of your application) before starting your updates. This should "purge" your endpoint. When update is done, update your load balancer again and put back the server in it.
Cloud concept is infrastructure as code: this could be easily scripted and included in you deployment / update procedure.
Another solution would be to use Traffic Manager. It could give you additional option to manage your endpoints (It might be a bit oversized for 2 VM / endpoints).
Last solution is to migrate to a PaaS solution where all this kind of features are already available (Deployment Slot).
Hoping this will help.
Best regards
My application has a load balancer which distributes payloads to two test servers. I need to test the load balancing using jmeter. Do I need to have an instance of jmeter run on all these servers to test this? How can I test this efficiently using jmeter? I also have to take out one of the test servers during load and see the response as well. Please advise how to proceed.
You should test the configuration you will be using in production, i.e. use single endpoint (which is the loadbalancer) instead of direct backend node hosts. You should not be running JMeter on any of the backend machines to avoid mutual interference. If single JMeter instance will not be enough to create the anticipated load and you will go for the distributed testing JMeter slave nodes (load generators) need to be installed outside the application under test.
Also if your Load Balancer in its turn has multiple IP addresses make sure your add DNS Cache Manager to your Test Plan to avoid IP caching on OS/JVM levels
Have a look to other questions, for example: JMeter - load balancing
And you'll certainly find some interesting tips in the
JMeter user manual
Apache Jmeter allows us to hit the server with simultaneous connections. On, the other hand I have 4 webservers - one acting as a load balancer and other 3 acting as a application server. So, i want to load test these servers at once to check its performance. Is there a way to load test a load balancer. Any tools that will be helpful to carry out. I will edit this question further with more information. For time being could someone point out a starting point.
Ramp your normal app-test as usual (through the load balancer).
Eventually, you'll get high response times. If you see your application servers are running fine, then it's (probably*) your load balancer that's the issue. If the application servers are falling over, then you don't need to worry about the performance of your load balancer- it's not the bottleneck.
*obviously, there could be other problems, eg, simple network throughput. But you should be able to tell what's going on with some simple monitoring.
Yes you can make a load test on your load balancer Apache server.
Target your http requests to the load balancer apache server. I'm assuming the LB will distribute the load evenly through the 3 backend servers. After the test you evaluate the response times. Are they good? Great.
If they aren't good you can make a second test targeting the 3 backend servers (don't let the requests pass by the LB). If the response times are better now then you know your LB is the problem, otherwise you may need to add more backend servers or optimize your applications (i guess this is what you'll do).