I am looking for a way to assert that the average time taken for certain HTTP requests is below an upper limit value:
I can add a duration assert to the HTTP request, but a spike in the duration for whatever reason would fail the assertion.
I can also compute these averages manually, given that I can access the individual request times, but I was hoping there might be an easier way to do it.
Is there any other way?
My way for this problem is to use short timeout (ms) for my HTTP request, in your case choose your limit as timeouts. Then every request with higher response time will count as error.
Related
This is the sort of traffic pattern I'm consistently seeing.
I understand that RPS roughly equals number of users/(response time + sleep time), hence my RPS will be roughly flat if my number of users and my response times are increasing at a similar rate (I'm using 0 sleep time).
I also understand that you can't help me debug the underlying system whose response time is increasing! That's another thread I'll be pursuing separately. The increasing response time is not a Locust issue.
My question is how can I get Locust to ignore response time, in order to produce a constantly increasing RPS? I would like to take response time out of the equation entirely so that RPS is proportional to number of users.
(Why do I want to do this? In order to effectively load test my particular system.)
An individual Locust user is syncronous/sequential and cannot ”ignore response times” any more than any other Python program can ”ignore the time spent executing a line of code”
But you can use wait_time = constant_pacing(seconds_per_iteration) to ensure a fixed iteration time for each user https://docs.locust.io/en/stable/writing-a-locustfile.html#wait-time-attribute
Or wait_time = constant_pacing(1/iterations_per_second) if you prefer.
For a ”global” version of the same type of wait, use https://github.com/SvenskaSpel/locust-plugins/blob/master/examples/constant_total_ips_ex.py
Make sure your user count is high enough, as none of these methods can launch additional users/concurrent requests.
You may also want to have a look at https://github.com/locustio/locust/wiki/FAQ#increase-my-request-raterps
Building on cyberwiz's answer, you can't make the individual Locust users ignore response time. Each has made a request and can't do anything else until it gets a response. With ever increasing response times, all you can do is make Locust spawn more and more users. You'd need to run in distributed mode and add more workers who can spawn more users. You can specify a higher user count and maybe even a higher hatch rate, depending on the behavior you're trying to achieve.
I ran a JMeter test for 193 samples
where I could see my average response time as 5915ms and Throghput as 1.19832.
I just want to know how are they exactly related
All the answers are in JMeter Glossary
Elapsed time. JMeter measures the elapsed time from just before sending the request to just after the last response has been received.
Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time).
The relationship is: higher response time - lower throughput and vice versa.
You can use charts like Transactions per Second for throughput and Response Times Over Time for response times to get them plotted on your test timeline and Composite Graph to put them together. This way you will be able to track the trends.
All 3 charts can be installed using JMeter Plugins Manager
TL;DR
No, but yes.
Both aren't related directly, but when increasing Throughput, it will probably affect server response time due to load/stress on server.
If there are timeout errors response time will probably increase.
But for validation or firewall errors - response time will probably decrease.
There's a long explanation in JMeter archive, last is using Disney to demonstrate:
Think of your last trip to disney or your favorite amusement park. Lets define capacity of the ride to be the number of people that can sit on the ride per turn (think roller coaster). Throughput will be the number of people that exit the ride per unit of time. Lets define service time the the amount of time you get to sit on the ride. Lets define response time or latency to be your time queuing for the ride (dead time) plus service time.
In terms of load/Performance testing. Throughput and Response times are inversely proportional. i.e
With increase in response time throughput should decrease.
With increase in Throughput response time should decrease.
You can get more detailed definitions in this blog:
https://nirajrules.wordpress.com/2009/09/17/measuring-performance-response-vs-latency-vs-throughput-vs-load-vs-scalability-vs-stress-vs-robustness/
Throughout increases to some extent and remains stable when all the resources becomes busy. Now, if user requests increases further at this point response time would increase. But if response time increase is only because of internal queuing then due to the fact that system is taking more requests in at the same time response time is also increasing, throughout doesn't change. When queues are full more requests should fail. If response increase is due to some delay in processing or serving the request, for example running a query on database then due to the fact that system is not accepting more request and at the same time response time is also increasing, consequently throughout would drop.
Just a general explaination.
Respose Time : It is the time calculated when user send the request till request gets finnished.
Throughput : It is server property that number of transaction or request can be made during certain amount of time. here 1.19832 /minute means server cand hadle 1.19832 sample per minute.
As Respose Time increses Throughput increases.
How is this possible that in Jmetetr increasing number of users (threads) in my test did not changed the latency (Response time)?
I got the same latency for 100 threads and for 300 threads.
Latency is the difference between the time when a request was sent and time when the response has started to be received.
As per JMeter Glossary
JMeter measures the latency from just before sending the
request to just after the first response has been received. Thus the
time includes all the processing needed to assemble the request as
well as assembling the first part of the response, which in general
will be longer than one byte. Protocol analysers (such as Wireshark)
measure the time when bytes are actually sent/received over the
interface. The JMeter time should be closer to that which is
experienced by a browser or other application client.
Response time (= Sample time = Load time = Elapsed time) is a difference between the time when the request was sent and time when the response has been fully received.
As per JMeter Glossary
JMeter measures the elapsed time from just before sending the request
to just after the last response has been received. JMeter does not
include the time needed to render the response, nor does JMeter
process any client code, for example, Javascript.
So Response time always >= latency.
So it is possible that you may have same Latency for 100 and 300 threads but Response time will be different or increased.
If you have stable network connectivity between JMeter and Application Under Test it is expected that Latency wouldn't change not matter how many threads you kick off. It is "pure" network metric which tells how long did it take for the request to reach to the server.
Check out How to Analyze the Results of a Load Test article to see the impact of Latency for the end user
I'm trying to understand a significant performance increase in my Jmeter test.
In a multi-tenancy database environment, I have a single RESTful service test containing a Thread Group with a single HTTP Request sampler posting an XML payload. The XML payload is then evaluated via stored procedures, and a response is received stating if the claim was qualified. I run this test from a .bat file (non-gui mode) in an Apache 7 environment with a single JVM running.
Test Thread Group Properties
# of Threads: ${__P(test.threads,200)}
Ramp-Up Period: ${__P(test.rampup,1)}
Loop Count: Forever
Delay Thread: Enabled
Scheduler: Enabled
Duration: ${__P(test.duration,1800)}
HTTP Request
Method: POST
https://serverName:port/database/.../${__P(tenant,1111)}/Claim/${__property(contractId)}
When I duplicate the HTTP Request sampler within the TG and change the tenant ID within the URL, for some reason the performance seems to increase by > 55%. (i.e., the # of claims/second is increased by 55%) It appears the test did not fail, so I cannot attribute the performance increase to an increased error rate.
I would have expected an increase if I had enabled another JVM to let the Load Balancer perform optimization, but this is not the case. (still using only 1 JVM)
HTTP Request 1
https://serverName:port/database/.../${__P(tenant,1111)}/Claim/${__property(contractId)}
HTTP Request 2
https://serverName:port/database/.../${__P(tenant,2222)}/Claim/${__property(contractId)}
The theory going around here is that Jmeter generates a workload at a higher rate for multiple requests than for a single request. I'm skeptical, but haven't found anything "solid" to support my skepticism.
Is this theory true? If so, why would two HTTP Requests increase the performance?
In short: it's OK.
Longer version:
Here is how JMeter works:
JMeter starts all the threads during ramp-up period
Each thread starts executing samplers upside down (or according to the Logic Controllers)
When request doesn't have more samplers to execute and no more loops to iterate it's being shut down.
So how does number of virtual users correlate with the "performance". When you increase virtual users number of requests number for a load test it affects Throughput
Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time).
So if you increase load on well-behaved system throughput should increase by the same factor or linearly.
When you increase load but throughput does not increase, such situation is known as "saturation point" when you get the maximum performance from the system. Further load increasing will lead to throughput going down.
References:
Apache JMeter Glossary
An extended Glossary version
And how do you messure your performance? According to your "theory" your messurements includes jmeter overhead and this would be wrong. More over, is the response the same for both cases? What I mean, is the backend doing the same work work both cases?
Maybe first request returns different output then the other one. Maybe it is more expensive to generate output in one of the request. That is why you will notice "incresed" performance as normally you would do N x Heavy task in X seconds, and in second case G x heavy tasks + H X light tasks in the same time where G < N/2 - more requests in the same time? Sure! Incresed perfomance? Nope.
So to completly investigate what is happening, you need to review your measurement method. I would start with comparing the the actual time for both requests.
I use JMeter to test my webapp application, I have aggregate graph with some score values, but I actually don't know what they mean...
Aggregate graph shows for example:
average
median
min
max
I don't know about what refer that values.
For what refer 90% line?
I also don't know what's the unit of throughput per second (bytes?).
Anybody knows?
JMeter documentation shows only general information about reports and listeners.
This link contains a helpful explanation for Jmeter Usuage, Results, Tips, Considerations and Deployment.
Good Luck!
Throughput - means number of requests per one second. So if two users open your website at the same time throughput will be 2/s - 2 requests in one second.
How it can be useful: check your website analytics and you see number of hosts and hits per one day. Throughput stands for hits per day. If analytics shows 200 000 hits per day this means: 200 000 / 86400 (seconds in one day) = 2,31 hits/s.
Average - the average of response time. I think you know what is response time - it's time between sending request and getting response from server. To get the average response time you should sum all samplers response time and devide to number of samplers. Sampler means user, request, hit, the meaning is the same.
Min - the minimal response time in all samplers. Differently we may say the fastest response.
Max - opposite of Min, the slowest response.
The throughput is generally measured in requests/second in things Jmeter.
As far as knowing what requests are within the 90% line, not really a way to do it with this listener. This is representing aggregate information, so it only reports on information about all the tests, not specific results.
For some different methods and ideas on getting useful information out of the responses, take a look at this jmeter wiki on log analysis.
If you don't already have it, jmeter plugins has a lot of useful controllers and listeners that can make understanding the results easier as well.