I have a webtest in visual studio, which login to a website and does few clicks. I wanted to simulate this for 2000 users without having those many logins in a csv file or a database. Configured my webtest to run the test for 30 minutes by stepping 10 users every 10 seconds, its running fine for like 2 minutes and then failing after that with multiple errors. Is this possible at all ? If so how can i do this ?
A load test is intended to do what is stated in your question but there is too little detail in the question to give a good answer. Adding 10 users every 10 seconds means that after 2 minutes there have been 2*60/10 = 12 increments, so 10(users) * 2(minutes) * 60(seconds per minute) / 10(step duration) = 120 users. If it ran for the full 30 minutes then there would be 10*30*60/10 = 1800 users. Microsoft provide recommendations for how many users can be run by one computer and when to use additional agent computers. These are just starting points and do not take the details of the individual tests into account.
Rather than jumping in with a 2000 user test it may be better to start more slowly and establish, for a modest number of users that, the test suite works and also that the web site works. I suggest rerunning the test incrementing by 10 users every minute. Then after 2 minutes there will be 10*2*60/10 = 120 users. If that test works then decrease the step interval to (perhaps) 20 seconds.
The question does not state what errors are being generated. The actual errors need to be analyzed to determine the nature and location of the problem. It may be that a limit is being reached in some part of the server, the network or the computer running the test. When the limiting component is found then you know what needs changing.
Related
I want to ask a question regarding of how to perform my load test scenarios. I have multiple threat groups that each one executes a scenario (get,post,put requests) with a different number of threats on each scenario. Each scenario takes an X (unknown) of time to be execute always depending of how many requests it has. What i want is to run them all together and hold the load (keep threats active) for a specific amount of time.
Let me give an example:
Threat Group 1 - v. users 75
Threat Group 2 - v. users 50
Threat Group 3 - v. users 30
I want to execute all threat groups together and keep the load for 30 mins. The problem that i have its lets say that threat group 1 it takes only 2 mins to be executed, threat group 2 takes only 3 minutes and threat group 3 takes only 1 minute. I can not have the Infinite option tick as i have only specific number of data that i can use. For the first scenario i have 75 users (data) that if they execute the scenario once they can not execute it again thus i need 75 more (data) which i dont have.
Is there any way to do that? Is it feasible to keep threats alive in each threat group for specific amount of time? Or something that will execute the requests in a specific amount of time that will cover the whole duration? (30 mins)
Setting up the parameters in threat groups like the picture below when each scenario ends it shuts down its threats so i will never have lets say 30 v users to run in parallel for a specific amount of time.
The below graph shows that Scenario 1 (pink) ended its threats faster than Scenario 2 (ref). Here the settings are the following:
Threats in each scenario 5
Ramp-up 10 secs
Duration 90 secs
If I understand correctly you are constrainded by data and you have one data point for one loop per user . So I will try to manipulate rampup and set it to 1800 and loop = 1 for every thread group. Uncheck "specify thread lifetime". Jmeter will run new thread e.g for TG_1 every 1800/75 second. Whole test will run 1800sec + time for loop for last thread. If have to tink if this load profile is appropriate for your application.
As I can understand your query, you have data for say 75 users, the data cannot be repeated. And you need to hold the load for 30 minutes.
In simple words, I can say that you cannot hold the load for this much time unless you have more data, or you can repeat data.
If 75 users are taking only 1 minute or even 5 minutes, this means that you have not added human reaction time. To add that, you can simply use timers. Which will delay the execution of sampler.
If your first thread group executes 75 requests in 2 minutes and you want it to be executed in 30 minutes the only way of achieving this is making JMeter 15x times slower. JMeter provides a variety of Timers and each of them allows introducing delays between requests mainly to simulate real user "think time"
75 requests in 2 minutes is approx 37 requests per minute and if you need to make it 15x times slower you need to execute ~2.4 requests per minute.
So you can add i.e. Constant Throughput Timer to your first thread group and configure it like:
this way you will get 72 requests in 30 minutes.
Similarly you can slow down 2 remaining thread groups
Try with Ultimate Thread group plugin
Use different ultimate thread group and change the threads as per your requirement but keep other parameters same as shown below.
So, your threads will vary but other condition probably will remain same. Like Initial delay, startup time, holdloadfor sec and shutdown.
You can use plugin manager to download the jpgc#stardard set for all the plugin provided by jpgc. Then, you will get option to add jpgc ultimate thread group from Threads(Users) or from the same place we add normal/standard threadgroup.
I want to stresstest a site using Jmeter, right now i am using WAMP and widnows.
What would be the best stresstesting settings for this? this is a page that will have alot of users, 100k users / day+
100k users per day is not that "lot" in my opinion:
100 000 / 24 == 4166.67 users per hour
4167 / 60 == 69.44 users per minute
69 / 60 = 1.15 users per second
The normal way of testing the application is starting with either 1 or expected amount of virtual users and gradually increase the load until one of the following conditions is met:
response time goes above acceptable level
application starts consuming > 80-90% of underlying hardware resources (CPU, RAM, Disk and/or Network IO)
application starts returning errors and the amount of errors exceeds threshold
Also I would recommend to test the following scenarios:
load test - put your application under anticipated load for short period to see how does performance metrics look like
stress test - determine the maximum amount of concurrent users that your application can handle keeping reasonable response times. You can also determine breaking point, i.e. when application starts returning errors or stops responding and whether it can recover
soak test - basically the same as point 1, but the load should sustain for a longer period, i.e. several hours or if time allows - several days. It will help to identify memory leaks if any
I have a WebAPI service that I put together to test throughput hosted in Azure. I have it set up to call Task.Delay with a configurable number (IE webservice/api/endpoint?delay=500). When I run against the endpoint via Fiddler, everything works as expected, delays, etc.
I created a Load Test using VS Enterprise and used some of my free cloud load testing minutes to slam it with 500 concurrent users over 2 minutes. After multiple runs of the load test, it says the average test time is roughly 1.64 seconds. I have turned off think times for the test.
When I run my request in Fiddler concurrently with the Load test, I am seeing sub-second responses, even when spamming the execute button. My load test is doing effectively the same thing and getting 1.64 second response times.
What am I missing?
Code running in my unit test (which is then called for my load test):
var client = new HttpClient { BaseAddress = new Uri(CloudServiceUrl) };
var response = client.GetAsync($"{AuthAsyncTestUri}&bankSimTime={bankDelay}&databaseSimTime={databaseDelay}");
AuthAsyncTestUri is the endpoint for my cloud-hosted service.
There are several delay(), sleep(), pause(), etc methods available to a process. These methods cause the thread (or possible the program or process for some of them) to pause execution. Calling them from code used in a load test is not recommended, see the bottom of page 187 of the Visual Studio Performance Testing Quick Reference Guide (Version 3.6).
Visual Studio load tests do not have one thread per virtual user. Each operating system thread runs many virtual users. On a four-core computer I have seen a load test using four threads for the virtual users.
Suppose a load test is running on a four-core computer and Visual Studio starts four threads to execute the test cases. Suppose one virtual user calls sleep() or similar. That will suspend that thread, leaving three threads available to execute other virtual user activity. Suppose that four virtual users call sleep() or similar at approximately the same time. That will stop all four threads and no virtual users will be able to execute.
Responding to the following comment that was added to the question
I did try running it with a 5 user load, and saw average test times of less than 500 ms, which match what I see in my Fiddler requests. I'm still trying to figure out why the time goes up dramatically for the 500 user test while staying the same for Fiddler requests run in the middle of the 500 user test.
I think that this comment highlights the problem. At a low user load, the Visual Studio load test and the Fiddler test give similar times. At higher loads something between the load test and the server is limiting throughput and causing the slowdown. It would be worth examining the network route between the computer running the tests and the system being tested. Are there any slow segments on that path? Are there any segments that might see the load test as a denial of service attack and hence might slow down the traffic?
Running a test for as little as 2 minutes does not really show how the test runs. The details in the question do net tell how many tests started, how many finished and how many were abandoned at the end of the two minute run. It is possible that many tests cases were abandoned and that the average time of those that completed was 1.6 second.
If you have the results of the problem run then look at the "details" section of the results. Expand the slider below the image to include the whole run. Tick the option (top left corner) to highlight failing tests. I would expect to see a lot of red at the two minute mark for failing tests. However, the two minute run may be too short compared to the sampling interval (in the run settings) to see much.
Running a first test at 500 users tells you very little. It tells you either that the system copes with that load or that it does not. You need to run the test at several different user loads. Then you start to learn where the boundary between working and not working lies. Hence I recommend using a stepped load.
I believe you need at least one more test run to understand what is happening. I suggest doing a run as follows. Set a one minute cool-down period. Set a stepped load: start at 5 users as you know that that works. Increment by 1 user every two seconds until 100 users. That will take 190 seconds. Run for about another minute at that 100 user load. Total of 4 minutes 10 seconds. Call it 4 minutes. Adding in the one minute cool down makes (5 minutes) x (100 VU) = 500 VUM, which is a small portion of the free minutes per month. After the run look at the graphs of average test times. If all is OK on that test then you could try another that ramps up more quickly to say 500 users.
Is that possible to simulate 1000 to 10000 user to jmeter?
If so how?
in 1 thread group only test 1k user:
Thread user = 1000,
Ramp up = 10 ,is this means it will test 1000 user in 10sec?,
Loopcount = 1, is there a disadvantage if I put 10 here ,is this means 1000x10 user = it will simulate 10k user?
Is 1 thread group can simulate 1000 concurrent user?
I could have written it as comment, but it went out of characters.
definitions given by #Quality-Expert are correct but his understanding of your test setup is wrong.
10 loop count doesnt mean 1000*10
It means at a given moment only 1000 concurrent users will be present but they will perform same action 10 times. It doesn't mean 10000 users.
Rampup is 10 doesnt mean 1000 users for 10 seconds
It means 100 users will come online in 1 second and next 100 users will online in next second i.e. in 2 seconds 200 users will come online and this is how 1000 users will come online in 10 seconds.
If you dont provide any test duration test will end their itself, it wont continue after threads have finished their task. If you want to load test with 1000 users then provide some test duration i.e. 10 mintues. Thus test will run with 1000 concurrent users for 10 min. Rampup is just warm up time for system under load test so that it wont face sudden load of 1000 users.
About 10000 load, use need to use distributed set of machines/client of Jmeter which will generate that load or use cloud load testing tools.
Number of Threads- The maximum number of users you want to run.
Ramp-Up Period - Defines how long it takes for JMeter to ramp up from zero users to X number of threads. It is in terms of seconds.
Loop Count - Defines how many times you want each user to run your script.
Other thing is that No. of user depends on your machine CPU , RAM , Physical memory. As far as I know using normal system you can user 250-500 max. users.
If you want to do testing like 1k,10k users then you will have to use cloud systems like Neoload.
As per your given example , you are right about if you put thread user = 1000 & ramp up = 10 then it will run test till 10 seconds for 1000 users.
Let me know if you have still any confusion.
Answer for the above question in below screen, in Jmeter manual its different which says threads/ramp-up period= user per seconds
I've recently started working in Jmeter and I have run a test with 40 virtual users. I've also used loadosophia plugin that uploads results to loadosophia. But I've few doubts about numbers I've got in the summary report in loadosophia.
I've run with 40 virtual users with 3 seconds as ramp up period but in loadosophia report it shows Max virtual users=17 and Average virtual users = 6.5 but Transaction count is 40 which actually should be virtual users. Does this indicate there were maximum of 17 virtual users were concurrent ?
Test run duration it shows 3 seconds, but I can see Maximum response time is 5078 milliseconds which is(5.7 seconds) how can it be test can be completed in 3 seconds while it shows some requests took more than 5 seconds to send response? (I am assuming it is not considering ramp up period which I gave 3 seconds, Am I right?)
Thanks in advance.
Currently your scenario looks as:
Start with 1 user (which will immediately start test actions)
Each second add 13 users (each of them will start test actions as soon as the thread is started)
When thread finished it's job it exits.
So there is no way to say exactly what will be the load at the "peak" where all threads which are active at the moment are hitting the application under test.
Not too sure regarding loadosophia, but if you expect 40 concurrent virtual users the only way to reach this load is providing enough loops and adding a Constant Throughput Timer to orchestrate the load and limit (or ramp-up) it to the value, you set in timer.
Another option is using a Synchronizing Timer which allows to "release" threads when specified amount is reached.