We are planning implementing Microbenchmark tests for our java based springboot project for which we are planning to leverage JMH.
We are in need of validating Latency and Bandwidth metric and not sure how to fetch these from JMH.
Can someone suggest a sample code or reference?
First of all, JMH measures throughput, not bandwidth.
Here's the example of measuring costs of Spring Boot-based application start-up: https://github.com/stsypanov/spring-boot-benchmark
And here are various examples of Spring-related measurements with JMH: https://github.com/stsypanov/spring-benchmark (e.g. this benchmark allows to measure costs of getting a new prototype-scoped bean from application context. As of your question if you want to measure latency (i.e. average execution time of benchmarked method) you use the mentioned benchmark as is. If you want to measure throughput then just change
#BenchmarkMode(Mode.AverageTime)
#OutputTimeUnit(TimeUnit.MICROSECONDS)
to
#BenchmarkMode(Mode.Throughput)
#OutputTimeUnit(TimeUnit.SECONDS)
The second annotation set tells JMH to print how many times the method can be called within a second.
Related
We would like to use Istio for our workload in a production environment. The required CPU resources are documented here: https://istio.io/latest/docs/ops/deployment/performance-and-scalability/
The documentation uses a load test scenario consisting of 1000 requests per second going through envoy proxies (among other parameters). We do have a much lower number of requests, and of course it is important to optimize CPU resources in order to optimize the cost of our solution.
So, the question is: can we reduce the CPU assignment to the envoy proxies proportionally to the reduction of parameters in our scenario? There is a minimum threshold for CPU assignment? I have found nothing about this in the documentation. Of course, we would like to keep a reasonable small latency added by the proxy (similar to the 2.65 ms in the documented scenario).
We can perform some load testing for our solution once we have Istio running; but the CPU additional resources consumed by Istio will be one of the relevant points in deciding whether to go with it or not. So some initial idea about this would be wonderful.
We need in our system to calculate and visualize a couple of business metrics like:
total number of transactions processed over last configured time interval
average processing time over last configured time interval
max processing time over last configured time interval
min processing time over last configured time interval
I have to expose these metrics somehow from SpringBoot application.
I checked whether it is possible to calculate this type of metrics at the application level (Spring boot) using the Micrometer library built into the Spring Boot Actuator. Unfortunately, I don't see that it allows using Meters to calculate the average value or the minimum value of particular method execution (with configured time interval)
The use of Prometheus also doesn't seem to be the best idea because it works on the pull-based principle. It seems to me that this may render the results inaccurate and delayed because of scraping intervals.
My last idea is to write each transaction processing time to InfluxDB or a similar DB and then, using queries get the results I need (business metrics). However, I am worried about the efficiency of this solution as it introduces additional time to each business transaction
What do you think about it? Am I right about the limitations of the Micrometer? Does the idea from influxDB sound reasonable? Maybe another way to approach this problem?
Are there any clear ideas that define the Scalability test? I have designed Load, Stress, Spike and Soak tests using JMeter Ultimate Thread Group but, i have not any idea about Scalability test differs from these tests. How to design a good scalability test with ultimate thread group in Jmeter for maximum user count is equal to 500.
As per the Wikipedia article on Scalability testing:
Scalability testing, is the testing of a software application to measure its capability to scale up or scale out in terms of any of its non-functional capability.
So basically you can use the same approach as for the Stress Testing, something like this:
then you need to pay attention to the following charts/KPIs:
Active Threads Over Time - to show number of active virtual users
Transactions per Second - to show the system throughput
Charts of system resources consumption - to show usage of CPU, RAM, etc.
Ideally the charts should be more or less the same/linear, i.e. if you increase the load by factor of 2x the throughput should increase by the same factor and resource consumption should increase proportionally.
If the charts are not equal/similar/proportional - then at some point the system won't be able to keep threads/transactions per second ratio and this will indicate the bottleneck
Why we are comparing Performance test results with base line if we already have SLA's?
How they will be related-
For example:
Transaction response time in main test is 3 seconds
SLA for same transaction is 5 second
Baseline for that transaction was 2 seconds
How to compare these?
If time is over SLA - you have a critical production issue that need to be address.
If time is over baseline - your server suffer from performance degradation and it need to be analyse,but in lesser importance
Read more in testingperformance:
Any user action where the response time seems to be higher than expected can be traced, monitored and checked to determine if their are any inefficiencies.
As the workload is increased, the performance tester can look to see how the response times of transactions deviate from the baseline as the workload increases.
This is a difficult question to answer - are you the recipient of an SLA (as in your system uses an external system with an SLA) or do you have to guarantee an SLA?
Typically people use "baseline" to mean the application as it is now, running in typical conditions and under typical load.
Typically, a response time SLA includes upper limits on load, or some kind of commercial ladder - guaranteeing a response time for unlimited traffic is often impossible without additional financial resources.
If your first performance test suggests that the actual response time is higher than baseline, it suggests that either you disagree about "typical" conditions, or that you've exceeded those typical conditions, or that the application's performance has deteriorated since the baseline was established. That's important information.
In general terms, response times and load do not have a linear relationship - if response time is 1 second with 100 users, it's usually not 10 seconds with 1000 users. Instead, response time tends to rise very slowly with load until you hit a bottleneck, at which point it rises very steeply.
I typically use performance testing to explore those bottlenecks, so I can decide how they fit with my desired performance characteristics, and work out how to move that bottleneck further away.
It's also worth noting that most systems have multiple bottlenecks, and the slowest element determines the overall performance characteristics. So even if you have an SLA for 5 second transactions in one part of your architecture, there may be other parts that are slower (or reach their bottleneck sooner).
So, why do you compare your load tests with base lines, even if you have an SLA?
Make sure that the baseline is still valid.
Make sure you understand the overall performance characteristics and can exceed the SLA in other parts of the system.
Verify you can reach the SLA
We are re-implementing(yes from scratch) a web application which is currently in production. We have decided to start doing some performance tests on the new app, to get some early information of the capabilities.
As the old application is currently in production and has a good performance we would like to extract some performance parameters, and then use this parameters as a reference or base goal of the performance of the new application.
Which do you think are the most relevant performance parameters we should be obtaining from the current production application?
Thanks!
Determine what pages are used the most.
Measure a latency histogram for the total time it takes to answer the request. Don't just measure the mean, measure a histogram.
From the histogram you can see how many % of requests have which latency in milliseconds. You can choose to key performance indicators by takes the values for 50% and 95%. This will tell you the average latency and the worst latency (for the worst 10% of requests).
Those two numbers alone will bring you great confidence regarding the experience your users will have.
Throughput does not matter for users, but for capacity planning.
I also recommend that you track the performance values over time and review them twice a year.
Just in case you need an HTTP client, there is weighttp, a multi-threaded client written by the guys from Lighttpd.
It has the same syntax used by ApacheBench, but weighttp lets you use several client worker threads (AB is single-threaded so it cannot saturate a modern SMP Web server).
The answer of "usr" is valid, but you can as well record the minnimum, average and maximum latencies (that's useful to see in which range they play). Here is a public-domain C program to automate all this on a given concurrency range.
Disclamer: I am involved in the development of this project.