I am trying to figure out which timer to use for my loadtests in order to simulate a gradual growth in traffic towards the website.
I had a look ad the Gaussian Random Timer:
To delay every user request for random amount of time use Gaussian
Random Timer with most of the time intervals happening near a specific
value.
and the Poisson random timer:
To pause each and every thread request for random amount of time use
Poisson Random Timer with most of the time intervals occurring close a
specific value.
taken from this source.
Now I don't really understand what's the difference between the two. They both apply a random delay that is more likely to be close to a specific value. So what am I missing? How to they differ in practice?
The difference is in the algorithm used to generate random values:
Poisson is based on this:
http://en.wikipedia.org/wiki/Poisson_distribution
http://www.johndcook.com/blog/2010/06/14/generating-poisson-random-values/
Gaussian uses :
java.util.Random#nextGaussian()
Both add to Constant Delay Offset the value of a random number generated based on either Poisson or Gaussian.
The difference is in underlying algorythm, check the following links for details
Normal (Gaussian) distribution
Poisson Distribution
I would also recommend reading A Comprehensive Guide to Using JMeter Timers article for exhaustive information on JMeter timers.
Related
I'm new to metric monitoring.
If we want to record the duration of the requests, I think we should use gauge, but in practise, someone would use histogram.
for example, in grpc-ecosystem/go-grpc-prometheus, they prefer to use histogram to record duration. Are there agreed best practices for the use of metric types? Or it is just their own preference.
// ServerMetrics represents a collection of metrics to be registered on a
// Prometheus metrics registry for a gRPC server.
type ServerMetrics struct {
serverStartedCounter *prom.CounterVec
serverHandledCounter *prom.CounterVec
serverStreamMsgReceived *prom.CounterVec
serverStreamMsgSent *prom.CounterVec
serverHandledHistogramEnabled bool
serverHandledHistogramOpts prom.HistogramOpts
serverHandledHistogram *prom.HistogramVec
}
Thanks~
I am new to this but let me try to answer your question. So take my answer with a grain of salt or maybe someone with experience in using metrics to observe their systems jumps in.
as stated in https://prometheus.io/docs/concepts/metric_types/
A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
So if your goal would be to display the current value (duration time of requests) you could use a gauge. But I think the goal of using metrics is to find problems within your system or generate alerts if and when certain vaules aren't in a predefined range or getting a performance value (like the Apdex score) for your system.
From https://prometheus.io/docs/concepts/metric_types/#histogram
Use the histogram_quantile() function to calculate quantiles from histograms or even aggregations of histograms. A histogram is also suitable to calculate an Apdex score.
From https://en.wikipedia.org/wiki/Apdex
Apdex (Application Performance Index) is an open standard developed by an alliance of companies for measuring performance of software applications in computing. Its purpose is to convert measurements into insights about user satisfaction, by specifying a uniform way to analyze and report on the degree to which measured performance meets user expectations.
Read up on Quantiles and the calculations in histograms and summaries https://prometheus.io/docs/practices/histograms/#quantiles
Two rules of thumb:
If you need to aggregate, choose histograms.
Otherwise, choose a histogram if you have an idea of the range and distribution of values that will be observed. Choose a summary if you need an accurate quantile, no matter what the range and distribution of the values is.
Or like Adam Woodbeck in his book "Network programming with Go" said:
The general advice is to use summaries when you don’t know the range of expected values, but I’d advise you to use histograms whenever possible
so that you can aggregate histograms on the metrics server.
The main difference between gauge and histogram metric types in Prometheus is that Prometheus captures only a single (last) value of the gauge metric when it scrapes the target exposing the metric, while histogram captures all the metric values by incrementing the corresponding histogram bucket.
For example, if request duration is measured for frequently requested endpoint and Prometheus is set up to scrape your app every 30 seconds (e.g. scrape_interval: 30s in scrape_configs), then the Prometheus will scrape only a single duration for the last request every 30 seconds when the duration is stored in a Gauge metric. All the previous measurements for the request duration are lost.
On the other hand, any number of request duration measurement are registered in Histogram metric, and this doesn't depend on the interval between scrapes of your app. Later the Histogram metric allows obtaining the distribution of request durations on an arbitrary time range.
Prometheus histograms have some issues though:
You need to choose the number and the boundaries of histogram buckets, so they provide good accuracy for observing the distribution of the measured metric. This isn't a trivial task, since you may not know in advance the real distribution of the metric.
If the number of buckets are changed or their boundaries are changed for some measurement, then the histogram_quantile() function returns invalid results over such a measurement.
To big number of buckets per each histogram may result in high cardinality issues, since each bucket in the histogram creates a separate time series.
P.S. these issues are addressed in VcitoriaMetrics histograms (I'm the core developer of VictoriaMetrics).
As valyala suggest, the main difference is that histogram aggregates data, so you would take advantage of prometheus statistics engine over all registered samples (minimum, maximum, average, quantiles, etc.).
A gauge is more used to measure for example "wind velocity", "queue size", or any other kind of "instant data" where it is not so important to ignore old related samples of it as you want to know current picture.
Using gauges for "duration of the requests" would require very small scrape periods to be accurate, which is not practical even if your rate is not very high (if your scrape period is less than your application reception rate, you will ignore data). So, in summary, don't use gauges. Histogram fits much better your needs.
In JMeter, we have multiple timers - for instance Uniform Standard Timer, Gaussian Random timer etc.
While researching my query - I found various books and blog which tell me
How to add the timers in JMeter
What is the internal formula/logic between the timers.
I am somehow confused, which one to use? and when?
For instance, if I am trying to wait for the user to log in, which timer is more appropriate? Uniform? or Gaussian?
As per A Comprehensive Guide to Using JMeter Timers
Uniform Random Timer
The Uniform Random Timer pauses the thread by a factor of:
The next pseudorandom uniformly-distributed value in range between 0.0 (inclusive) and 1.0 (exclusive)
Multiplied by “Random Delay Maximum”
Plus “Constant Delay Offset”
Gaussian Random Timer
A Gaussian Random Timer calculates the thread delay time using an approach like a Uniform Random Timer does, but instead of a uniformly-distributed pseudorandom value in 0.0 - 0.9 range, the normal (a.k.a. Gaussian) distribution is being used as the first argument for the formula.
There are several algorithms for generating normally distributed values, in JMeter Marsaglia polar method is used which takes the next 2 random values U and V in -1 to 1 range until the S = U2 + V2 > 1 condition is met. Once S is defined it is used in the formula
to return the next pseudorandom Gaussian (“normally”) distributed value. The first time the method is called it returns X, the second time it will return Y, the third time it starts over and will return the new X, etc.
With regards to "which timer to use and when" - there is no "good" answer which fits all the cases, the timers you've mentioned are used to simulate think time as real users don't hammer the application non-stop, they need some time to "think" between operations. So the decision is up to you:
on one hand real users perform different delays between operations so it would be good to randomize this delay a little bit if you want the test to be as much realistic as possible
on the other hand load test needs to be repeatable so it makes sense to go for Constant Timers to avoid any random factor impacting the test results
maybe a good idea would be using Uniform or Gaussian random timer initially in order to mimic real users and then for regression testing purposes switch to the Constant Timer for results repeatability purposes
There's a table at the bottom of
https://www.tensorflow.org/performance/performance_guide#optimizing_for_cpu that talks about images per second and step time, in the context of performance in deep learning.
How does one compute images/second and step time?
When training in Tensorflow using the Estimator API, there's a number reported as global_step/sec. Is this the same thing? If so, does that take into account the time required to process the input pipeline, or just the time it takes to do the forward pass through the model?
global_step/sec means how many steps per second your tensorflow model is doing. A step is usually a minibatch. So the inverse of global_step/sec is your step time and batch_size * global_step/sec is your number of images per second.
Because these numbers are throughput numbers computed on the steady state of your model training they include the input pipeline (i.e. if your input pipeline cannot produce X minibatches per second then you necessarily have to run less than X global_step/sec).
A hardware sensor is sampled precisely (precise period of sampling) using a real-time unit. However, the time value is not sent to the database together with the sampled value. Instead, time of insertion of the record to the database is stored for the sample in the database. The DATETIME type is used, and the GETDATE() function is used to get current time (Microsoft SQL Server).
How can I reconstruct the precise sampling times?
As the sampling interval is (should be) 60 seconds exactly, there was no need earlier for more precise solution. (This is an old solution, third party, with a lot of historical samples. This way it is not possible to fix the design.)
For processing of the samples, I need to reconstruct the correct time instances for the samples. There is no problem with shifting the time of the whole sequence (that is, it does not matter whether the start time is rather off, not absolute). On the other hand, the sampling interval should be detected as precisely as possible. I also cannot be sure, that the sampling interval was exactly 60 seconds (as mentioned above). I also cannot be sure, that the sampling interval was really constant (say, slight differences based on temperature of the device).
When processing the samples, I want to get:
start time
the sampling interval
the sequence o the sample values
When reconstructing the samples, I need to convert it back to tuples:
time of the sample
value of the sample
Because of that, for the sequence with n samples, the time of the last sample should be equal to start_time + sampling_interval * (n - 1), and it should be reasonably close to the original end time stored in the database.
Think in terms of the stored sample times slightly oscillate with respect to the real sample-times (the constant delay between the sampling and the insertion into the database is not a problem here).
I was thinking about calculating the mean value and the corrected standard deviation for the interval calculated from the previous and current sample times.
Discontinuity detection: If the calculated interval is greater than 3 sigma off the mean value, I would consider it a discontinuity of the sampled curve (say, the machine is switched off, or any outer event lead to missing samples. In the case, I want to start with processing a new sequence. (The sampling frequency could also be changed.)
Is there any well known approach to the problem. If yes, can you point me to the article(s)? Or can you give me the name or acronym of the algorithm?
+1 to looking at the difference sequence. We can model the difference sequence as the sum of a low frequency truth (the true rate of the samples, slowly varying over time) and high frequency noise (the random delay to get the sample into the database). You want a low-pass filter to remove the latter.
For my SIMULINK model I need to generate a signal that takes the values 1 or 0. To generate it I need to draw a number from a an exponential distribution and use this number as the time the signal stays in 0. Once this time has passed, I have to draw a new number from the exponential distribution and use this number as the time the signal stays in 1, and the repeat the process until the end of the simulation. As a SIMULINK newbie I'm quite puzzled by this problem and would appreciate any suggestions on how to solve it.
You've got a couple of choices.
In MATLAB, you can generate all samples in advance (i.e. before running the simulation) and use them to create a suitable signal, then use that as an input into the model (using the From Workspace block).
Or, if you need to do the sampling at each time step, then you have to write an S-Function, using the random number in the mdlGetTimeOfNextVarHit method. There is an example of doing something very similar on the Goddard Consulting web site called Square Wave with Jitter.