I have a simple JMeter throughput test that spawns 20 threads and each thread executes a simple SQL query against a database. I have set a ramp up time of 10 seconds and a total test time of 70 seconds.
When I execute the test in non-GUI mode I see the following summary output:
summary + 1 in 0.1s = 7.4/s Avg: 135 Min: 135 Max: 135 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary + 137501 in 28.5s = 4831.0/s Avg: 3 Min: 1 Max: 614 Err: 0 (0.00%) Active: 20 Started: 20 Finished: 0
summary = 137502 in 29s = 4796.9/s Avg: 3 Min: 1 Max: 614 Err: 0 (0.00%)
summary + 171000 in 30s = 5703.8/s Avg: 3 Min: 1 Max: 519 Err: 0 (0.00%) Active: 20 Started: 20 Finished: 0
summary = 308502 in 59s = 5260.8/s Avg: 3 Min: 1 Max: 614 Err: 0 (0.00%)
summary + 61016 in 11.5s = 5309.0/s Avg: 3 Min: 1 Max: 518 Err: 0 (0.00%) Active: 0 Started: 20 Finished: 20
summary = 369518 in 70.1s = 5268.9/s Avg: 3 Min: 1 Max: 614 Err: 0 (0.00%)
As you can see that the throughput is low in the first 30 seconds but picks up later. I understand that this could be due to the threads starting and other system components warming up.
Is there a way I can exclude "X" seconds worth of starting numbers from the final calculation. I have used some custom performance tests where I always excluded the first "X" seconds until the system reached a steady state before measuring the output.
Is there anyway I can do that in JMeter?
Option 1:
JMeter result file is basically CSV file having the following format:
timeStamp,elapsed,label,responseCode,responseMessage,threadName,dataType,success,bytes,grpThreads,allThreads,Latency
1440671556283,1799,HTTP Request,200,OK,Thread Group 1-1,text,true,1591,1,1,1798
where first column is sample start time (in milliseconds since Unix Time)
If you need to omit first 30 seconds add 30000 to the first sampler timestamp, locate the closest row and delete all the rows from 1st to that one.
Option 2:
You can use Synchronizing Timer to ensure that load starts with 20 threads from the early beginning.
You can use jmeter-plugins to generate graphs
http://jmeter-plugins.org/wiki/GraphsGeneratorListener/
And fill in start offset field.
Read this for using generation:
https://www.ubik-ingenierie.com/blog/automatically-generating-nice-graphs-at-end-of-your-load-test-with-apache-jmeter-and-jmeter-plugins/
Related
I have an RTO problem that I want to solve for multiple simulated timesteps with some time-depended parameters. However, I'm struggling with the run-time and noticed that the total system time is relatively large compared to the actual solve time. I was therefore trying to reduce the total parse time, as all the equations remain the same - yet "only" the values of some parameters change with time. A simple example below:
#parameters from simulation
demand = 100
#do RTO
from gekko import GEKKO
# first, create the model
m = GEKKO(remote=False)
# declare additional decision variables
m.u = m.Var(lb=5, ub=25)
m.v = m.Var(lb=0, ub=100)
m.w = m.Var(lb=0, ub=50)
m.b = m.Var(lb=0, ub=1, integer=True)
m.demand = m.Param(demand)
# now add the objective and the constraints
m.Minimize((1-0.8)*m.u*m.b+(1-0.9)*m.v+(1-0.7)*m.w)
m.Equation(m.u*m.b >= 10)
m.Equation(m.u*m.b + m.v + m.w == m.demand)
m.options.SOLVER=1
m.options.DIAGLEVEL = 1
m.solve()
then I capture the results, execute them in the simulation and move on to the next timestep. Now I could just, execute all code above again - with updated parameters (let's say the demand is now 110). But this results in the before mentioned long run-time (the RTO problem needs to be build from scratch every time, while only some parameters change). So I thought the following could work:
m.demand.VALUE = 110
m.solve()
While this does work. It doesn't seem to improve the run-time (total parse time is still relatively long). Below are the display outputs of the actual problem.
First time solving the RTO problem.
----------------------------------------------------------------
APMonitor, Version 1.0.0
APMonitor Optimization Suite
----------------------------------------------------------------
Called files( 55 )
files: overrides.dbs does not exist
Run id : 2022y11m03d13h18m21.919s
COMMAND LINE ARGUMENTS
coldstart: 0
imode : 3
dbs_read : T
dbs_write: T
specs : T
rto selected
Called files( 35 )
READ info FILE FOR VARIABLE DEFINITION: gk_model6.info
SS MODEL INIT 0
Parsing model file gk_model6.apm
Read model file (sec): 0.6602
Initialize constants (sec): 0.
Determine model size (sec): 0.4170999999999999
Allocate memory (sec): 0.
Parse and store model (sec): 0.45140000000000025
--------- APM Model Size ------------
Each time step contains
Objects : 247
Constants : 0
Variables : 752
Intermediates: 249
Connections : 741
Equations : 745
Residuals : 496
Error checking (sec): 0.17809999999999993
Compile equations (sec): 1.9933000000000003
Check for uninitialized intermediates (sec): 0.
------------------------------------------------------
Total Parse Time (sec): 3.7062
SS MODEL INIT 1
SS MODEL INIT 2
SS MODEL INIT 3
SS MODEL INIT 4
Called files( 31 )
READ info FILE FOR PROBLEM DEFINITION: gk_model6.info
Called files( 6 )
Files(6): File Read rto.t0 F
files: rto.t0 does not exist
Called files( 51 )
Read DBS File defaults.dbs
files: defaults.dbs does not exist
Called files( 51 )
Read DBS File gk_model6.dbs
files: gk_model6.dbs does not exist
Called files( 51 )
Read DBS File measurements.dbs
Called files( 51 )
Read DBS File overrides.dbs
files: overrides.dbs does not exist
Number of state variables: 1240
Number of total equations: - 989
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 251
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter: 1 I: 0 Tm: 1.20 NLPi: 45 Dpth: 0 Lvs: 3 Obj: 9.86E-02 Gap: NaN
--Integer Solution: 2.32E-01 Lowest Leaf: 9.86E-02 Gap: 1.34E-01
Iter: 2 I: 0 Tm: 0.06 NLPi: 4 Dpth: 1 Lvs: 2 Obj: 2.32E-01 Gap: 1.34E-01
Iter: 3 I: 0 Tm: 0.23 NLPi: 6 Dpth: 1 Lvs: 2 Obj: 2.16E-01 Gap: 1.34E-01
Iter: 4 I: 0 Tm: 0.44 NLPi: 12 Dpth: 1 Lvs: 3 Obj: 1.60E-01 Gap: 1.34E-01
--Integer Solution: 2.32E-01 Lowest Leaf: 1.60E-01 Gap: 7.21E-02
Iter: 5 I: 0 Tm: 0.20 NLPi: 6 Dpth: 2 Lvs: 2 Obj: 1.01E+00 Gap: 7.21E-02
--Integer Solution: 2.06E-01 Lowest Leaf: 2.06E-01 Gap: 0.00E+00
Iter: 6 I: 0 Tm: 0.20 NLPi: 5 Dpth: 2 Lvs: 2 Obj: 2.06E-01 Gap: 0.00E+00
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 2.3522999999999996 sec
Objective : 0.20599966381706797
Successful solution
---------------------------------------------------
Called files( 2 )
Called files( 52 )
WRITE dbs FILE
Called files( 56 )
WRITE json FILE
Timer # 1 6.57/ 1 = 6.57 Total system time
Timer # 2 2.35/ 1 = 2.35 Total solve time
Timer # 3 0.01/ 156 = 0.00 Objective Calc: apm_p
Timer # 4 0.01/ 78 = 0.00 Objective Grad: apm_g
Timer # 5 0.01/ 156 = 0.00 Constraint Calc: apm_c
Timer # 6 0.00/ 0 = 0.00 Sparsity: apm_s
Timer # 7 0.00/ 0 = 0.00 1st Deriv #1: apm_a1
Timer # 8 0.01/ 78 = 0.00 1st Deriv #2: apm_a2
Timer # 9 0.42/ 1 = 0.42 Custom Init: apm_custom_init
Timer # 10 0.00/ 1 = 0.00 Mode: apm_node_res::case 0
Timer # 11 0.00/ 1 = 0.00 Mode: apm_node_res::case 1
Timer # 12 0.02/ 1 = 0.02 Mode: apm_node_res::case 2
Timer # 13 0.00/ 1 = 0.00 Mode: apm_node_res::case 3
Timer # 14 0.17/ 317 = 0.00 Mode: apm_node_res::case 4
Timer # 15 0.72/ 156 = 0.00 Mode: apm_node_res::case 5
Timer # 16 0.00/ 0 = 0.00 Mode: apm_node_res::case 6
Timer # 17 0.01/ 78 = 0.00 Base 1st Deriv: apm_jacobian
Timer # 18 0.00/ 78 = 0.00 Base 1st Deriv: apm_condensed_jacobian
Timer # 19 0.00/ 1 = 0.00 Non-zeros: apm_nnz
Timer # 20 0.00/ 0 = 0.00 Count: Division by zero
Timer # 21 0.00/ 0 = 0.00 Count: Argument of LOG10 negative
Timer # 22 0.00/ 0 = 0.00 Count: Argument of LOG negative
Timer # 23 0.00/ 0 = 0.00 Count: Argument of SQRT negative
Timer # 24 0.00/ 0 = 0.00 Count: Argument of ASIN illegal
Timer # 25 0.00/ 0 = 0.00 Count: Argument of ACOS illegal
Timer # 26 0.00/ 1 = 0.00 Extract sparsity: apm_sparsity
Timer # 27 0.00/ 13 = 0.00 Variable ordering: apm_var_order
Timer # 28 0.00/ 1 = 0.00 Condensed sparsity
Timer # 29 0.00/ 0 = 0.00 Hessian Non-zeros
Timer # 30 0.00/ 1 = 0.00 Differentials
Timer # 31 0.00/ 0 = 0.00 Hessian Calculation
Timer # 32 0.00/ 0 = 0.00 Extract Hessian
Timer # 33 0.00/ 1 = 0.00 Base 1st Deriv: apm_jac_order
Timer # 34 0.01/ 1 = 0.01 Solver Setup
Timer # 35 1.39/ 1 = 1.39 Solver Solution
Timer # 36 0.00/ 167 = 0.00 Number of Variables
Timer # 37 0.01/ 84 = 0.00 Number of Equations
Timer # 38 0.01/ 14 = 0.00 File Read/Write
Timer # 39 0.00/ 0 = 0.00 Dynamic Init A
Timer # 40 0.00/ 0 = 0.00 Dynamic Init B
Timer # 41 0.00/ 0 = 0.00 Dynamic Init C
Timer # 42 0.66/ 1 = 0.66 Init: Read APM File
Timer # 43 0.00/ 1 = 0.00 Init: Parse Constants
Timer # 44 0.42/ 1 = 0.42 Init: Model Sizing
Timer # 45 0.00/ 1 = 0.00 Init: Allocate Memory
Timer # 46 0.45/ 1 = 0.45 Init: Parse Model
Timer # 47 0.18/ 1 = 0.18 Init: Check for Duplicates
Timer # 48 1.99/ 1 = 1.99 Init: Compile Equations
Timer # 49 0.00/ 1 = 0.00 Init: Check Uninitialized
Timer # 50 0.01/ 1257 = 0.00 Evaluate Expression Once
Timer # 51 0.00/ 0 = 0.00 Sensitivity Analysis: LU Factorization
Timer # 52 0.00/ 0 = 0.00 Sensitivity Analysis: Gauss Elimination
Timer # 53 0.00/ 0 = 0.00 Sensitivity Analysis: Total Time
Updating one parameter and only calling m.solve() again - is shown in the simple problem above.
----------------------------------------------------------------
APMonitor, Version 1.0.0
APMonitor Optimization Suite
----------------------------------------------------------------
Called files( 55 )
Called files( 55 )
files: overrides.dbs does not exist
Run id : 2022y11m03d13h18m28.729s
COMMAND LINE ARGUMENTS
coldstart: 0
imode : 3
dbs_read : T
dbs_write: T
specs : T
rto selected
Called files( 35 )
READ info FILE FOR VARIABLE DEFINITION: gk_model6.info
SS MODEL INIT 0
Parsing model file gk_model6.apm
Read model file (sec): 0.6901
Initialize constants (sec): 0.
Determine model size (sec): 0.4546999999999999
Allocate memory (sec): 0.
Parse and store model (sec): 0.2824000000000002
--------- APM Model Size ------------
Each time step contains
Objects : 247
Constants : 0
Variables : 752
Intermediates: 249
Connections : 741
Equations : 745
Residuals : 496
Error checking (sec): 0.16720000000000002
Compile equations (sec): 2.0142999999999995
Check for uninitialized intermediates (sec): 0.
------------------------------------------------------
Total Parse Time (sec): 3.6097
SS MODEL INIT 1
SS MODEL INIT 2
SS MODEL INIT 3
SS MODEL INIT 4
Called files( 31 )
READ info FILE FOR PROBLEM DEFINITION: gk_model6.info
Called files( 6 )
Files(6): File Read rto.t0 T
Called files( 51 )
Read DBS File defaults.dbs
files: defaults.dbs does not exist
Called files( 51 )
Read DBS File gk_model6.dbs
Called files( 51 )
Read DBS File measurements.dbs
Called files( 51 )
Read DBS File overrides.dbs
files: overrides.dbs does not exist
Number of state variables: 1240
Number of total equations: - 989
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 251
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter: 1 I: 0 Tm: 0.26 NLPi: 7 Dpth: 0 Lvs: 3 Obj: 9.35E-02 Gap: NaN
--Integer Solution: 1.21E-01 Lowest Leaf: 9.35E-02 Gap: 2.71E-02
Iter: 2 I: 0 Tm: 0.22 NLPi: 5 Dpth: 1 Lvs: 2 Obj: 1.21E-01 Gap: 2.71E-02
--Integer Solution: 1.21E-01 Lowest Leaf: 9.35E-02 Gap: 2.71E-02
Iter: 3 I: 0 Tm: 0.32 NLPi: 10 Dpth: 1 Lvs: 1 Obj: 1.03E+00 Gap: 2.71E-02
Iter: 4 I: 0 Tm: 0.25 NLPi: 8 Dpth: 1 Lvs: 1 Obj: 1.20E-01 Gap: 2.71E-02
--Integer Solution: 1.21E-01 Lowest Leaf: 1.86E-01 Gap: -6.58E-02
Iter: 5 I: 0 Tm: 0.37 NLPi: 15 Dpth: 2 Lvs: 1 Obj: 1.86E-01 Gap: -6.58E-02
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 1.4365000000000006 sec
Objective : 0.12065435497282542
Successful solution
---------------------------------------------------
Called files( 2 )
Called files( 52 )
WRITE dbs FILE
Called files( 56 )
WRITE json FILE
Timer # 1 5.64/ 1 = 5.64 Total system time
Timer # 2 1.44/ 1 = 1.44 Total solve time
Timer # 3 0.00/ 91 = 0.00 Objective Calc: apm_p
Timer # 4 0.00/ 45 = 0.00 Objective Grad: apm_g
Timer # 5 0.01/ 91 = 0.00 Constraint Calc: apm_c
Timer # 6 0.00/ 0 = 0.00 Sparsity: apm_s
Timer # 7 0.00/ 0 = 0.00 1st Deriv #1: apm_a1
Timer # 8 0.00/ 45 = 0.00 1st Deriv #2: apm_a2
Timer # 9 0.44/ 1 = 0.44 Custom Init: apm_custom_init
Timer # 10 0.00/ 1 = 0.00 Mode: apm_node_res::case 0
Timer # 11 0.00/ 1 = 0.00 Mode: apm_node_res::case 1
Timer # 12 0.01/ 1 = 0.01 Mode: apm_node_res::case 2
Timer # 13 0.00/ 1 = 0.00 Mode: apm_node_res::case 3
Timer # 14 0.10/ 187 = 0.00 Mode: apm_node_res::case 4
Timer # 15 0.46/ 90 = 0.01 Mode: apm_node_res::case 5
Timer # 16 0.00/ 0 = 0.00 Mode: apm_node_res::case 6
Timer # 17 0.00/ 45 = 0.00 Base 1st Deriv: apm_jacobian
Timer # 18 0.00/ 45 = 0.00 Base 1st Deriv: apm_condensed_jacobian
Timer # 19 0.00/ 1 = 0.00 Non-zeros: apm_nnz
Timer # 20 0.00/ 0 = 0.00 Count: Division by zero
Timer # 21 0.00/ 0 = 0.00 Count: Argument of LOG10 negative
Timer # 22 0.00/ 0 = 0.00 Count: Argument of LOG negative
Timer # 23 0.00/ 0 = 0.00 Count: Argument of SQRT negative
Timer # 24 0.00/ 0 = 0.00 Count: Argument of ASIN illegal
Timer # 25 0.00/ 0 = 0.00 Count: Argument of ACOS illegal
Timer # 26 0.00/ 1 = 0.00 Extract sparsity: apm_sparsity
Timer # 27 0.00/ 13 = 0.00 Variable ordering: apm_var_order
Timer # 28 0.00/ 1 = 0.00 Condensed sparsity
Timer # 29 0.00/ 0 = 0.00 Hessian Non-zeros
Timer # 30 0.01/ 1 = 0.01 Differentials
Timer # 31 0.00/ 0 = 0.00 Hessian Calculation
Timer # 32 0.00/ 0 = 0.00 Extract Hessian
Timer # 33 0.00/ 1 = 0.00 Base 1st Deriv: apm_jac_order
Timer # 34 0.01/ 1 = 0.01 Solver Setup
Timer # 35 0.84/ 1 = 0.84 Solver Solution
Timer # 36 0.00/ 102 = 0.00 Number of Variables
Timer # 37 0.00/ 51 = 0.00 Number of Equations
Timer # 38 0.12/ 14 = 0.01 File Read/Write
Timer # 39 0.00/ 0 = 0.00 Dynamic Init A
Timer # 40 0.00/ 0 = 0.00 Dynamic Init B
Timer # 41 0.00/ 0 = 0.00 Dynamic Init C
Timer # 42 0.69/ 1 = 0.69 Init: Read APM File
Timer # 43 0.00/ 1 = 0.00 Init: Parse Constants
Timer # 44 0.45/ 1 = 0.45 Init: Model Sizing
Timer # 45 0.00/ 1 = 0.00 Init: Allocate Memory
Timer # 46 0.28/ 1 = 0.28 Init: Parse Model
Timer # 47 0.17/ 1 = 0.17 Init: Check for Duplicates
Timer # 48 2.01/ 1 = 2.01 Init: Compile Equations
Timer # 49 0.00/ 1 = 0.00 Init: Check Uninitialized
Timer # 50 0.00/ 505 = 0.00 Evaluate Expression Once
Timer # 51 0.00/ 0 = 0.00 Sensitivity Analysis: LU Factorization
Timer # 52 0.00/ 0 = 0.00 Sensitivity Analysis: Gauss Elimination
Timer # 53 0.00/ 0 = 0.00 Sensitivity Analysis: Total Time
Many thanks in advance for your ideas.
Here are a few ideas to improve the compile time speed:
If the problem is a simulation, it can be run with IMODE=7 to compile the model once and run through all the timesteps sequentially. This is often the fastest option for simulation.
If the problem has degrees of freedom (find optimal parameter to minimize objective) then it is also possible to set up the problem as IMODE=6 to compile the model once and it solves all time steps simultaneously.
The model re-compiles every time m.solve() is called. Keeping a compiled version of the model is built into the REPLAY parameter, but it has little documentation and is built for exactly reenacting a sequence from historical data. If you'd like the option to keep a compiled version of the model for the next run, please add it as a feature request in GitHub.
Using the simple model shows that the model re-compile time doesn't change significantly between runs with different values of demand.
from gekko import GEKKO
from numpy import random
import numpy as np
import matplotlib.pyplot as plt
import time
# first, create the model
m = GEKKO(remote=False)
# declare additional decision variables
m.u = m.Var(lb=5, ub=25)
m.v = m.Var(lb=0, ub=100)
m.w = m.Var(lb=0, ub=50)
m.b = m.Var(lb=0, ub=1, integer=True)
m.demand = m.Param()
# now add the objective and the constraints
m.Minimize((1-0.8)*m.u*m.b+(1-0.9)*m.v+(1-0.7)*m.w)
m.Equation(m.u*m.b >= 10)
m.Equation(m.u*m.b + m.v + m.w == m.demand)
m.options.SOLVER=1
m.options.DIAGLEVEL = 0
tt = [] # total time
ts = [] # solve time
for i in range(101):
start = time.time()
m.demand.value = 50+random.rand()*50
m.solve(disp=False)
tt.append(time.time()-start)
ts.append(m.options.SOLVETIME)
plt.figure(figsize=(8,4))
plt.plot(tt,label='Total Time')
plt.plot(ts,label='Solve Time')
plt.plot(np.array(tt)-np.array(ts),label='Compile Time')
plt.ylim([0,0.03]); plt.grid(); plt.ylabel('Time (sec)')
plt.savefig('timing.png',dpi=300)
plt.legend()
plt.show()
The compile and solve times are very fast for this simple problem. If you can post the full RTO problem, we can give more specific suggestions to improve solve time.
I want to make a load test on my API with 500k API calls.
I made a JMeter test that reads from a .csv file an access token and with that token make an API call.
Of course, from one computer I can't do that many requests so I implement distribute JMeter servers so I have on AWS one master and 500 servers that are the slave (t3.medium).
At the test, I set loops to be: 1000 (not so big number)
Ramp Up: 5 seconds. I want the test to last for 5 seconds. So a total load of 500k API calls to be made during a 5 seconds frame. It doesn't matter when the test starts, but I need to make all the requests from all the slave servers during this 5 seconds interval.
And here is the problem that the test always takes longer. For example, it takes like 3 minutes to make all the requests, and the average response from the server is around 50ms.
How can I configure JMeter or what should I use in order to guarantee that all 500k API calls are made during a 5sec interval?
This is an example output from the master:
Starting remote engines
Starting the test # Thu Nov 08 18:28:02 UTC 2018 (1541701682517)
summary + 1 in 00:00:00 = 4.8/s Avg: 106 Min: 106 Max: 106 Err: 0 (0.00%) Active: 76 Started: 24 Finished: 0
summary + 286890 in 00:00:27 = 10817.9/s Avg: 66 Min: 44 Max: 1363 Err: 0 (0.00%) Active: 2837 Started: 22851 Finished: 25632
summary = 286891 in 00:00:27 = 10732.9/s Avg: 66 Min: 44 Max: 1363 Err: 0 (0.00%)
summary + 40389 in 00:00:30 = 1344.8/s Avg: 67 Min: 45 Max: 1630 Err: 0 (0.00%) Active: 6607 Started: 27937 Finished: 24366
summary = 327280 in 00:00:57 = 5765.6/s Avg: 66 Min: 44 Max: 1630 Err: 0 (0.00%)
summary + 47986 in 00:00:30 = 1600.6/s Avg: 70 Min: 45 Max: 1618 Err: 0 (0.00%) Active: 2847 Started: 70360 Finished: 70549
summary = 375266 in 00:01:27 = 4326.1/s Avg: 67 Min: 44 Max: 1630 Err: 0 (0.00%)
summary + 8420 in 00:00:31 = 274.6/s Avg: 62 Min: 45 Max: 335 Err: 0 (0.00%) Active: 1231 Started: 77129 Finished: 78934
summary = 383686 in 00:01:57 = 3267.9/s Avg: 66 Min: 44 Max: 1630 Err: 0 (0.00%)
summary + 9173 in 00:00:30 = 305.1/s Avg: 59 Min: 45 Max: 1107 Err: 0 (0.00%) Active: 166 Started: 80976 Finished: 83846
summary = 392859 in 00:02:27 = 2663.8/s Avg: 66 Min: 44 Max: 1630 Err: 0 (0.00%)
Remote engines have been started
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 107131 in 00:01:11 = 1504.3/s Avg: 72 Min: 45 Max: 1230 Err: 0 (0.00%) Active: -484 Started: 86185 Finished: 88700
summary = 499990 in 00:03:39 = 2286.2/s Avg: 68 Min: 44 Max: 1630 Err: 0 (0.00%)
Tidying up remote # Thu Nov 08 18:31:41 UTC 2018 (1541701901996)
And this is how JMeter test looks like
There is one suitable option: Throughput Shaping Timer and Concurrency Thread Group combination. They can be put together via Schedule Feedback Function so JMeter will start extra threads if current amount is not enough to conduct the required load.
General recommendations:
Let's scale down a little bit, you need to ensure that 1 server can send 1000 requests in 5 seconds or 200 requests/second. It means that your server response time should be 5 milliseconds (including the time to establish the connection and requests to travel back and forth). As far as I can see minimum response time for your setup is 45 milliseconds and maximum is as high as 1,6 seconds. Therefore it looks like you will not be able to achieve the desired throughput using your current setup.
Now you need to identify the reason, here are some hints:
You have 5 seconds ramp-up, it means that last 200 users will be started on 5th second of your test, in any case it will take longer than 5 seconds
Amazon T3 medium is 2-core machine with 4 GB of RAM. Theoretically it should be enough to kick off 1000 users, however it depends on the size of request/response. Make sure JMeter engines are not overloaded using Amazon CloudWatch or JMeter PerfMon Plugin
Try running your test longer with a longer ramp-up time, it might give your application possibility to "warm-up" and you will be able to define the saturation point - the maximum number of users which your application can support without demonstrating performance degradation
I have a single Thread Group defined with 200 threads. I have a HTTP Request defined within the thread group and a Constant throughput timer defined within the HTTP Request. The Target throughput (samples/min) defined is 5400 samples/min (Since I want to achieve a throughput of 90 requests/second).
I am running the test in non-GUI mode. This is the summary report I see though:
Creating summariser <summary>
Created the tree successfully using Elasticsearch_Backup.jmx
Starting the test # Thu Jun 14 02:38:44 UTC 2018 (1528943924375)
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 187 in 00:00:46 = 4.1/s Avg: 34306 Min: 4 Max: 45427 Err: 13 (6.95%) Active: 200 Started: 200 Finished: 0
summary + 532 in 00:00:29 = 18.2/s Avg: 14309 Min: 3 Max: 53467 Err: 6 (1.13%) Active: 200 Started: 200 Finished: 0
summary = 719 in 00:01:15 = 9.6/s Avg: 19510 Min: 3 Max: 53467 Err: 19 (2.64%)
summary + 1318 in 00:00:30 = 43.8/s Avg: 4949 Min: 2 Max: 13973 Err: 10 (0.76%) Active: 200 Started: 200 Finished: 0
summary = 2037 in 00:01:45 = 19.4/s Avg: 10089 Min: 2 Max: 53467 Err: 29 (1.42%)
summary + 410 in 00:00:31 = 13.4/s Avg: 11060 Min: 4 Max: 27259 Err: 5 (1.22%) Active: 200 Started: 200 Finished: 0
summary = 2447 in 00:02:16 = 18.0/s Avg: 10251 Min: 2 Max: 53467 Err: 34 (1.39%)
summary + 300 in 00:01:00 = 5.0/s Avg: 23086 Min: 3 Max: 76712 Err: 4 (1.33%) Active: 200 Started: 200 Finished: 0
summary = 2747 in 00:03:16 = 14.0/s Avg: 11653 Min: 2 Max: 76712 Err: 38 (1.38%)
summary + 95 in 00:00:31 = 3.0/s Avg: 57490 Min: 3 Max: 93311 Err: 1 (1.05%) Active: 200 Started: 200 Finished: 0
summary = 2842 in 00:03:47 = 12.5/s Avg: 13185 Min: 2 Max: 93311 Err: 39 (1.37%)
summary + 84 in 00:00:28 = 3.0/s Avg: 76465 Min: 57500 Max: 119520 Err: 0 (0.00%) Active: 200 Started: 200 Finished: 0
summary = 2926 in 00:04:15 = 11.5/s Avg: 15002 Min: 2 Max: 119520 Err: 39 (1.33%)
summary + 98 in 00:00:30 = 3.3/s Avg: 62451 Min: 18075 Max: 121665 Err: 0 (0.00%) Active: 200 Started: 200 Finished: 0
summary = 3024 in 00:04:45 = 10.6/s Avg: 16539 Min: 2 Max: 121665 Err: 39 (1.29%)
summary + 204 in 00:00:30 = 6.8/s Avg: 47297 Min: 8554 Max: 152392 Err: 2 (0.98%) Active: 200 Started: 200 Finished: 0
summary = 3228 in 00:05:15 = 10.2/s Avg: 18483 Min: 2 Max: 152392 Err: 41 (1.27%)
summary + 723 in 00:00:30 = 24.1/s Avg: 10706 Min: 4 Max: 161663 Err: 2 (0.28%) Active: 200 Started: 200 Finished: 0
summary = 3951 in 00:05:45 = 11.4/s Avg: 17060 Min: 2 Max: 161663 Err: 43 (1.09%)
summary + 1089 in 00:00:30 = 36.2/s Avg: 5725 Min: 3 Max: 159356 Err: 3 (0.28%) Active: 200 Started: 200 Finished: 0
summary = 5040 in 00:06:15 = 13.4/s Avg: 14611 Min: 2 Max: 161663 Err: 46 (0.91%)
I dont see the requests/second go more than 43. Is something wrong with my configuration ?
Constant Throughput Timer can only pause threads in order to limit JMeter's throughput to the defined value. As per documentation>
Of course the throughput will be lower if the server is not capable of handling it, or if other timers or time-consuming test elements prevent it.
Looking into the Average Response Time which can be as high as > 1 minute my expectation is that your server simply cannot respond fast enough so you are not reaching the desired throughput due to some form of bottleneck.
If high response times are expected - you can try increasing number of threads in the Thread Group or even better switch to Concurrency Thread Group and Throughput Shaping Timer combination. These 2 test elements can be coupled via feedback function so JMeter will automatically kick off more threads if current amount is not enough to reach/maintain the desired throughput. You can install both elements using JMeter Plugins Manager
I'm struggling to understand how JMeter summariser works:
03:06:30 [debug] summary = 73930 in 1523s = 48.5/s Avg: 174 Min: 9 Max: 33133 Err: 4 (0.01%)
03:07:01 [debug] summary + 1573 in 78.2s = 20.1/s Avg: 142 Min: 10 Max: 2428 Err: 1 (0.06%) Active: 1600 Started: 1600 Finished: 0
03:07:01 [debug] summary = 75503 in 1555s = 48.6/s Avg: 173 Min: 9 Max: 33133 Err: 5 (0.01%)
03:07:30 [debug] summary + 1538 in 82.3s = 18.7/s Avg: 118 Min: 10 Max: 2704 Err: 0 (0.00%) Active: 1600 Started: 1600 Finished: 0
03:07:30 [debug] summary = 77041 in 1583s = 48.7/s Avg: 172 Min: 9 Max: 33133 Err: 5 (0.01%)
I have 73930 transactions executed in 1523s, which gives average of 48.5/s. Then two bunches of transactions are added on top of that:
1573 in 78.2s = 20.1/s
1538 in 82.3s = 18.7/s
Note the averages - 20.1/s and 18.7/s are below initial 48.5/s. I'd expect those to decrease overall statistics, but it goes up instead, by 0.2/s:
77041 in 1583s = 48.7/s
What I can see is: transactions are added as I'd expect (e.g. 73930 + 1573 = 75503), but how 1523s become 1555s and then 1583 is not clear to me.
Can someone explain that to me?
From the above given logs, it is clear that you have completed the execution of 73930 transactions at 03:06:30. Then 3111(1573+1538) transactions were executed by 03:07:30. So the extra 3111 transactions has taken a time of 60 sec. Which gets added to the original time 1523s. So 1523+60=1583.
It is not summing the time taken for the extra transactions as we sum the extra transactions to the original number of transactions(73930).
You can sum 73930+1573 and then get the total sum as 75503. But you should not sum the time as 1523s+78.2s. The time it is calculating is coming from the original time stamp.
I created a basic TCP server that reads incoming binary data in protocol buffer format, and writes a binary msg as response. I would like to benchmark the the roundtrip time.
I tried iperf, but could not make it send the same input file multiple times. Is there another benchmark tool than can send a binary input file repeatedly?
If you have access to a linux or unix machine1, you should use tcptrace. All you need to do is loop through your binary traffic test while capturing with wireshark or tcpdump file.
After you have that .pcap file2, analyze with tcptrace -xtraffic <pcap_filename>3. This will generate two text files, and the average RTT stats for all connections in that pcap are shown at the bottom of the one called traffic_stats.dat.
[mpenning#Bucksnort tcpperf]$ tcptrace -xtraffic willers.pcap
mod_traffic: characterizing traffic
1 arg remaining, starting with 'willers.pcap'
Ostermann's tcptrace -- version 6.6.1 -- Wed Nov 19, 2003
16522 packets seen, 16522 TCP packets traced
elapsed wallclock time: 0:00:00.200709, 82318 pkts/sec analyzed
trace file elapsed time: 0:03:21.754962
Dumping port statistics into file traffic_byport.dat
Dumping overall statistics into file traffic_stats.dat
Plotting performed at 15.000 second intervals
[mpenning#Bucksnort tcpperf]$
[mpenning#Bucksnort tcpperf]$ cat traffic_stats.dat
Overall Statistics over 201 seconds (0:03:21.754962):
4135308 ttl bytes sent, 20573.672 bytes/second
4135308 ttl non-rexmit bytes sent, 20573.672 bytes/second
0 ttl rexmit bytes sent, 0.000 bytes/second
16522 packets sent, 82.199 packets/second
200 connections opened, 0.995 conns/second
11 dupacks sent, 0.055 dupacks/second
0 rexmits sent, 0.000 rexmits/second
average RTT: 67.511 msecs <------------------
[mpenning#Bucksnort tcpperf]$
The .pcap file used in this example was a capture I generated when I looped through an expect script that pulled data from one of my servers. This was how I generated the loop...
#!/usr/bin/python
from subprocess import Popen, PIPE
import time
for ii in xrange(0,200):
# willers.exp is an expect script
Popen(['./willers.exp'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
time.sleep(1)
You can adjust the sleep time between loops based on your server's accept() performance and the duration of your tests.
END NOTES:
A Knoppix Live-CD will do
Filtered to only capture test traffic
tcptrace is capable of very detailed per-socket stats if you use other options...
================================
[mpenning#Bucksnort tcpperf]$ tcptrace -lr willers.pcap
1 arg remaining, starting with 'willers.pcap'
Ostermann's tcptrace -- version 6.6.1 -- Wed Nov 19, 2003
16522 packets seen, 16522 TCP packets traced
elapsed wallclock time: 0:00:00.080496, 205252 pkts/sec analyzed
trace file elapsed time: 0:03:21.754962
TCP connection info:
200 TCP connections traced:
TCP connection 1:
host c: myhost.local:44781
host d: willers.local:22
complete conn: RESET (SYNs: 2) (FINs: 1)
first packet: Tue May 31 22:52:24.154801 2011
last packet: Tue May 31 22:52:25.668430 2011
elapsed time: 0:00:01.513628
total packets: 73
filename: willers.pcap
c->d: d->c:
total packets: 34 total packets: 39
resets sent: 4 resets sent: 0
ack pkts sent: 29 ack pkts sent: 39
pure acks sent: 11 pure acks sent: 2
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 2512 unique bytes sent: 14336
actual data pkts: 17 actual data pkts: 36
actual data bytes: 2512 actual data bytes: 14336
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 17 pushed data pkts: 33
SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/0
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 6 adv wind scale: 1
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 792 bytes max segm size: 1448 bytes
min segm size: 16 bytes min segm size: 32 bytes
avg segm size: 147 bytes avg segm size: 398 bytes
max win adv: 40832 bytes max win adv: 66608 bytes
min win adv: 5888 bytes min win adv: 66608 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 14035 bytes avg win adv: 66608 bytes
initial window: 32 bytes initial window: 40 bytes
initial window: 1 pkts initial window: 1 pkts
ttl stream length: 2512 bytes ttl stream length: NA
missed data: 0 bytes missed data: NA
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 1.181 secs data xmit time: 1.236 secs
idletime max: 196.9 ms idletime max: 196.9 ms
throughput: 1660 Bps throughput: 9471 Bps
RTT samples: 18 RTT samples: 24
RTT min: 43.8 ms RTT min: 0.0 ms
RTT max: 142.5 ms RTT max: 7.2 ms
RTT avg: 68.5 ms RTT avg: 0.7 ms
RTT stdev: 35.8 ms RTT stdev: 1.6 ms
RTT from 3WHS: 80.8 ms RTT from 3WHS: 0.0 ms
RTT full_sz smpls: 1 RTT full_sz smpls: 3
RTT full_sz min: 142.5 ms RTT full_sz min: 0.0 ms
RTT full_sz max: 142.5 ms RTT full_sz max: 0.0 ms
RTT full_sz avg: 142.5 ms RTT full_sz avg: 0.0 ms
RTT full_sz stdev: 0.0 ms RTT full_sz stdev: 0.0 ms
post-loss acks: 0 post-loss acks: 0
segs cum acked: 0 segs cum acked: 9
duplicate acks: 0 duplicate acks: 1
triple dupacks: 0 triple dupacks: 0
max # retrans: 0 max # retrans: 0
min retr time: 0.0 ms min retr time: 0.0 ms
max retr time: 0.0 ms max retr time: 0.0 ms
avg retr time: 0.0 ms avg retr time: 0.0 ms
sdv retr time: 0.0 ms sdv retr time: 0.0 ms
================================
You can always stick a shell loop around a program like iperf. Also, assuming iperf can read from a file (thus stdin) or programs like ttcp, could allow a shell loop catting a file N times into iperf/ttcp.
If you want a program which sends a file, waits for your binary response, and then sends another copy of the file, you probably are going to need to code that yourself.
You will need to measure the time in the client application for a roundtrip time, or monitor the network traffic going from, and coming to, the client to get the complete time interval. Measuring the time at the server will exclude any kernel level delays in the server and all the network transmission times.
Note that TCP performance will go down as the load goes up. If you're going to test under heavy load, you need professional tools that can scale to thousands (or even millions in some cases) of new connection/second or concurrent established TCP connections.
I wrote an article about this on my blog (feel free to remove if this is considered advertisement, but I think it's relevant to this thread): http://synsynack.wordpress.com/2012/04/09/realistic-latency-measurement-in-the-application-layers
As a very simple highlevel tool netcat comes to mind ... so something like time (nc hostname 1234 < input.binary | head -c 100) assuming the response is 100 bytes long.