In this case, I have created several webtest scripts, and added them to a load test (distributed by expected use).
What I would like to do is send a user load (500 for example) where all users run at the same time, each user is given only a single script to run and complete, then the test is finished. One iteration for each user.
I am finding that iterations are not user based but test based, so only one user and test is completed when selecting a Test Iterations value of 1 for 500 users.
Is there a user based iteration setting or some other way to accomplish my intended test?
Thanks.
The test settings you have used are not at all clear from your question. However assuming you want to start 500 test cases at the same time and stop after they have completed then you can use the following.
In the properties of the scenario: Set the user load to constant and to 500 users. Also set maximum test iteration to 0 (meaning no maximum). I would also set the think time between iteration to much longer than you expect the test run to take; this setting may not be needed but it avoids unexpected behaviours.
In the properties of the run settings there are two possibilities.
Either (1) set the test iterations to 500.
Or (2) set the run duration to long enough for all 500 tests to complete, but shorter than the think time between iteration in the scenario.
Related
I have a web test where my requirements need a handful of different polling requests to be going on in the background. I have created a WebTestPlugin that looks for a certain context parameter to be set, and once it is, it kicks off a thread that just loops (every X seconds) firing off the configured request.
My issue is that this is not done in the context of the test, therefore the results (# of calls, duration, etc) is not part of the final report.
Is there a way to insert this data?
Rather than starting your own thread to run the background requests I suggest using the facilities of the load test. That way the results will be properly recorded. Another reason is that the threading regime of a load test is not specified by Microsoft and adding your own thread may cause issues.
You could have one scenario for the main test. Another scenario has one or more simple tests for the background polling activity. These tests could be set with a "think time between iterations" or with "test mix based on user pace" to achieve the required background rate. To get the background web tests starting at the correct time start the test with a constant load of 0 (zero) users and use a load test plugin that adjusts the number of users whenever needed. The plugin writes the required number into m_loadTest.Scenarios[N].CurrentLoad for a suitable N. This would probably be done in the Heartbeat plugin but potentially could be in any load test plugin. If may be that the TestFinished plugin can better detect when the number of users should increase.
I have a WebAPI service that I put together to test throughput hosted in Azure. I have it set up to call Task.Delay with a configurable number (IE webservice/api/endpoint?delay=500). When I run against the endpoint via Fiddler, everything works as expected, delays, etc.
I created a Load Test using VS Enterprise and used some of my free cloud load testing minutes to slam it with 500 concurrent users over 2 minutes. After multiple runs of the load test, it says the average test time is roughly 1.64 seconds. I have turned off think times for the test.
When I run my request in Fiddler concurrently with the Load test, I am seeing sub-second responses, even when spamming the execute button. My load test is doing effectively the same thing and getting 1.64 second response times.
What am I missing?
Code running in my unit test (which is then called for my load test):
var client = new HttpClient { BaseAddress = new Uri(CloudServiceUrl) };
var response = client.GetAsync($"{AuthAsyncTestUri}&bankSimTime={bankDelay}&databaseSimTime={databaseDelay}");
AuthAsyncTestUri is the endpoint for my cloud-hosted service.
There are several delay(), sleep(), pause(), etc methods available to a process. These methods cause the thread (or possible the program or process for some of them) to pause execution. Calling them from code used in a load test is not recommended, see the bottom of page 187 of the Visual Studio Performance Testing Quick Reference Guide (Version 3.6).
Visual Studio load tests do not have one thread per virtual user. Each operating system thread runs many virtual users. On a four-core computer I have seen a load test using four threads for the virtual users.
Suppose a load test is running on a four-core computer and Visual Studio starts four threads to execute the test cases. Suppose one virtual user calls sleep() or similar. That will suspend that thread, leaving three threads available to execute other virtual user activity. Suppose that four virtual users call sleep() or similar at approximately the same time. That will stop all four threads and no virtual users will be able to execute.
Responding to the following comment that was added to the question
I did try running it with a 5 user load, and saw average test times of less than 500 ms, which match what I see in my Fiddler requests. I'm still trying to figure out why the time goes up dramatically for the 500 user test while staying the same for Fiddler requests run in the middle of the 500 user test.
I think that this comment highlights the problem. At a low user load, the Visual Studio load test and the Fiddler test give similar times. At higher loads something between the load test and the server is limiting throughput and causing the slowdown. It would be worth examining the network route between the computer running the tests and the system being tested. Are there any slow segments on that path? Are there any segments that might see the load test as a denial of service attack and hence might slow down the traffic?
Running a test for as little as 2 minutes does not really show how the test runs. The details in the question do net tell how many tests started, how many finished and how many were abandoned at the end of the two minute run. It is possible that many tests cases were abandoned and that the average time of those that completed was 1.6 second.
If you have the results of the problem run then look at the "details" section of the results. Expand the slider below the image to include the whole run. Tick the option (top left corner) to highlight failing tests. I would expect to see a lot of red at the two minute mark for failing tests. However, the two minute run may be too short compared to the sampling interval (in the run settings) to see much.
Running a first test at 500 users tells you very little. It tells you either that the system copes with that load or that it does not. You need to run the test at several different user loads. Then you start to learn where the boundary between working and not working lies. Hence I recommend using a stepped load.
I believe you need at least one more test run to understand what is happening. I suggest doing a run as follows. Set a one minute cool-down period. Set a stepped load: start at 5 users as you know that that works. Increment by 1 user every two seconds until 100 users. That will take 190 seconds. Run for about another minute at that 100 user load. Total of 4 minutes 10 seconds. Call it 4 minutes. Adding in the one minute cool down makes (5 minutes) x (100 VU) = 500 VUM, which is a small portion of the free minutes per month. After the run look at the graphs of average test times. If all is OK on that test then you could try another that ramps up more quickly to say 500 users.
Writing a Web Performance Test for a process that will run for an undetermined time, and have to put a refresh command in a while that runs until the process state indicates it is done.
The refresh command consumes about 3 seconds. so do not want it running constantly in the loop. So, am trying to find a sleep/wait function to stop the execution between loops.
The only reference i've found is for Thread.Sleep which seems to do the job.
BUT, this method seems to also stop the test's timers. so, however many times the loop runs, and whatever the actual time taken by the process, the test report will only show the cumulative time of the refresh statements.
Is there another method that will not stop the test's timers?
If the refresh is in a loop within the Web Performance Test then set a suitable "think time" on the request. This will pause the test after the response is received. (Think times are normally used to simulate the time a person spends reading a web page and filling in forms etc before the next request is issued.)
Think times are set via the properties of the request. Think times (also reporting names) for all requests in a test can be viewed and modified using the "Set request detail" command accessed using the (rightmost) command icon in eth web test editor.
Think times can also be set or adjusted in the PreRequest method of a WebTestRequest plugin.
I have a JMeter Test Plan with many copies of almost exactly the same test. In each case there is a variable that is slightly different.
Here is the configuration:
There are two sets of user variables. There is a top level user variable list, that contains maximum_runs and there are Test Fragment level user variable lists with the User Defined Variable add_users, which goes up by 10 for each test case. users is a static 10.
I set maximum_runs to 100 and disable all but one Test Fragment. This gives me a number of samples = 100 for each Fragment. I enable a second Test Fragment and I still get 100 samples. But as soon as I enable the third Test Fragment my number of samples drops to 90. 4th, 80. But on the 5th one it goes right back up to 100 and the cycle starts over again. I don't see anything wrong with my math so I believe it to be something about how JMeter uses jexl2 or maybe variables are being changed due to the number of Fragments running? I really need to be able to run this with the same number of samples no matter how many Fragments are running. Ah, note, I have Run Thread Groups consecutively (i.e. run groups one at a time) in Test Plan checked.
I had a similar issue with one application. 1 out of 4 test components just would not go up more, than 50 percent of required users.
The problem was that component was a memory eater and when it reached the maximum heap it did not let the other threads in that component to ramp-up. But just a long shot.
I'm testing a large BizTalk system using Visual Studio Load Test. Load Test to pushes messages into MQ, these are picked up by BizTalk and then processed.
Rather than having the test finish (and all performance counters ending) as soon as Visual Studio has finished injecting messages to MQ, I want the test to end if and only if some condition is met (in my case if (SELECT COUNT(*) FROM BizTalkMsgBoxDb.dbo.Spool) == 4).
I can see a bunch of ways to run stuff after the test is complete, but no obvious way to extend the test and continue monitoring unless some user-defined exit condition is met.
Is this possible, or if not, does anyone have an idea for a good work-around/hack to achieve this?
You'll want to write a custom load test plugin. Details begin at this URL: http://msdn.microsoft.com/en-us/library/ms243153.aspx
The plugin can manipulate the scenario, extending the duration of the test until your condition is met.
I imagine you want to keep the load test running after queueing up a bunch of requests in order to continue to monitor the performance while the requests are processed. Although we can't control the load test duration, there is a way to achieve this.
Don't limit the test duration: Set the load test duration (or number of iterations) to a very large value -- larger than you anticipate (or know) it will take for the end condition to be satisfied.
Limit the scenario that queues up requests: In the load test scenario properties, in the Options section, set the Maximum Test Iterations so that the user load will drop to zero after sending the desired number of requests. If setting an iteration limit is not possible for some reason, you can instead write a load test plugin that sets the user load to zero in a specified scenario after a certain amount of test time has elapsed.
Check for end condition: Write a web test plugin that checks the database for your end condition. Attach this plugin to a new webtest in a new scenario and set Think Time Between Test Iterations on the scenario so that the test runs only as often as needed (60 seconds?). When the condition is reached, the plugin should write a predetermined value into the user context (the user context is accessible in the web test context as $LoadTestUserContext, and is only available in a load test, not when running a webtest standalone).
Abort the test: Write a load test plugin that looks for the flag value in the user context in the TestFinished event. When the value is found, the plugin calls LoadTest.Abort().
There is one minor disadvantage to this method: the test state is marked as Aborted in the results database.
At time of writing there is (still) no way to extend the duration of the test using a custom load test plugin, nor by having a virtual user type that refused to exit, nor by locking the close-down period of the test and preventing it from exiting that way.
The only way we managed to do something like this was to directly manipulate the LoadTest database and inject performance counter data in afterwards from log files, but this is neither smart nor recommended.
Oh well..