I want to test the performance of a REST API while simulating the actions performed by a user in the GUI. It needs to be able to perform a total of 4 requests in sequence, where request 2, 3 and 4 is based on information received in the first request (getting IDs, locations, etc, and then using those in request 2,3 and 4), and then store the results from each request.
Is there any testing frameworks that are capable of performing such actions programatically? Preferably based on JAVA, but not necessarily. I'm planning to run this for an estimation of 30 users. Any tips would be appreciated
The most popular Java-based test framework is REST-assured, for multithreading you can use either ExecutorService or Java Microbenchmark Harness
Alternatively you can go for a load testing tool, the most popular of the Java-based ones are:
Apache JMeter - where you can create API tests using GUI, see REST API Testing - How to Do it Right for details
Gatling - it's in Scala but runs in JVM, see API Load Testing With Gatling to get started
Related
Please could someone suggest:
I need to performance test a few APIs. Which of the below methods will be better way of conducting the tests (trying to understand the pros and cons of various ways to run JMeter tests):
Create Ultimate Concurrency Thread groups using only Jmeter.
Buy Blazemeter, add the Jmeter scripts, create scenario and run the tests
Run the JMeter tests using Maven plugin as part of the build.
Thanks,
N
Testing the performance of an API can be achieved by the above mentioned 3 ways.
Open Source: (Standalone JMeter/Taurus)
Cloud based solutions (Octoperf/Blazemeter)
If you are looking for an open source tool + you have your LG machine allocated + not gonna test with massive number of users - I would suggest to go with standalone JMeter/ taurus for your load testing solution.
If you are looking for an hassle free environment to manage all your test, LGs, reports, monitoring, etc then cloud based solutions like Blazemeter/Octoperf seems promising. This is going to cost you money and requires a port needs to be open for the solution to communicate with AUT.
For Scripting: You can use the same JMeter scripting in all 3 cases. Choosing a thread group requires what kinda test you wanna perform on your microsevrices.
In any case you will be using JMeter so points 2 and 3 don't really differ. The pro is that you don't have to pay. The con is that you need to manage your infrastructure yourself.
BlazeMeter provides easy way of launching a JMeter test so you don't have to worry about creating and configuring machines, installing JMeter, setting up distributed testing, real time monitoring and results comparison.
JMeter Maven Plugin makes the following processes easier:
installing JMeter
generating reporting dashboard
returning non-zero exit status code when acceptance criteria are not met
Trying to Performance test an application developed in OJET technology. Which tool/protocol should I use for scripting? I tried HTTP/Web protocol with Jmeter and Load Runner. But that doesn't capture all the requests and responses at the javascript/browser level. Hence I am facing issues in correlating the dynamic values during test design. Hence, scripts fail during the replay. Currently trying to do it with Truclient Web protocol as an alternative. But I need to know which tool/protocol should I use for scripting?
According to OJET looks like this is a web app generator.
If you choose to start with JMeter use post-processor such as regex to catch and save every value that is needed for as arg in the next request.
Don't be afraid of these dynamic values. Try to follow next articles to get the idea.
None of tools will provide you automatic correlation without issue. Nor LoadRunner, nor Jmeter. It is always tricky.
Ask more specific questions when you start facing issue.
Jmeter catch correlations
You need to implement real user using your application with 100% accuracy in terms of network footprint
Both JMeter and/or LoadRunner are not capable of executing client-side JavaScript, the options are in:
Implement these JavaScript-driven network calls using scripting (in JMeter it will be JSR223 Test Elements)
Use a real browser, LoadRunner's Truclient protocol is basically a headless web browser, in JMeter can be integrated with Selenium browser automation framework via WebDriver Sampler
With regards to "which protocol/tool" to use:
Implementing JavaScript calls manually will take extra effort, however your test will consume less resources (CPU, RAM, etc.)
Using real browsers will take less efforts, but the test will consume much more resources (something like 1 CPU core and 2 GB of RAM per user/ browser instance) and you won't have metrics like Connect Time, Latency, etc.
LoadRunner TruClient. This will handle all of the Javascript executions and dynamic elements related to session, state, date/time, object identifiers, ... You will still need to appropriately handle user input items.
I am new to Performance testing.
I want to Performance test my application that calls a 3rd party API (Transunion) for 2000 concurrent users using JMeter. The 3rd party API Transunion has limitation of up to 5 concurrent users maximum at a time.
How should I do Performance testing?
Thanks.
You should not be testing the 3rd-party API as this is not something you can efficiently control and even if you discover that it doesn't support 5 users but supports 1 - there is not much you will be able to do about it.
Your load test should focus solely on your application and your domain, all external stuff like banners, images, maps, videos, 3rd-party integrations should be excluded.
If the 3rd-party API is an integral part of your application you can use a Mock Object pattern to not to make the real call to the 3rd-party API but rather return a "dummy" response. If you cannot implement mocking in your application code you can use an external program like WireMock or HTTP API Mock or similar.
But be aware that the whole integrated system acts at the speed of its slowest component so if this 3rd-party integration is the essential part of your application - I don't see a lot of sense in performing the load test apart from concurrency testing like "what happens if from 2 to 5 users will do X at the same moment"
This is a classic Use Case for API simulations / remote mocks. They act as stand-ins for the real 3rd-party API dependencies that may be unavailable, impose rate limits/throttle calls, charge per transaction, etc. and they can also simulate slowness in such APIs, inject faults, and more to help with end-to-end integration or performance testing.
There is a choice of tools out there with different capabilities; you can write your own, too.
Disclaimer: I am involved with one such tool so I'll refrain from providing tool recommendations :-)
I have to test my rest API such that 100k API calls are made simultaneously(within 500ms).Any Idea how to simulate it?Utility to use?
I would suggest to use JMeter too, it enable your test to create multiple concurrent jmeter server. Just to be clear you can control multiple remote JMeter engines from a single JMeter client and replicate a test across many computers and thus simulate a larger load on the target server.
To be honest, your target is quite high (100k API calls simultaneously within 500ms), i.e. you'll need a lot of jmeter servers. When you create stress tests, there are not magical recipes, guides or manuals. Trial and error is a fundamental method of solving this kind of problems.
In my experience, I first try with few concurrent users and see how the server react. Then increase the number of concurrent users till to reach an intolerable performance decrease or, worst, a bottleneck .
http://jmeter.apache.org/usermanual/remote-test.html
You will obviously need a load testing tool which can be run in a distributed mode, i.e. 1 controller and X load generators executing the same test.
Grinder - scripts are written in some Python dialect
Apache JMeter - this guy doesn't require any specific knowledge, you can create tests using simple GUI
Tsung - is written in Erlang, known for capability to produce high loads even on low-end hardware.
See Open Source Load Testing Tools: Which One Should You Use? article for more information on above tools.
JMeter
The Apache JMeter™ application is open source software, a 100% pure
Java application designed to load test functional behavior and measure
performance. It was originally designed for testing Web Applications
but has since expanded to other test functions.
I would like to do integration testing on my Web API Controllers.
When the integration test starts the whole request/response pipeline of the Web API should be processed so its a real integration test.
I have read some blogs about non-InMemory-tests or InMemory tests. I need to know what is the difference and what of those approaches matches my above criteria?
I would really be glad about some explanations from people who really dealt with integration testing on Web API for self-hosting or IIS hosting (if there is a difference in testing...)
Not sure what you mean by non-in-memory testing but with integration testing involving an in-memory hosted web API, the requests are sent directly to the HttpServer, which is basically the first component to run in ASP.NET Web API pipeline. This means, the requests do not hit the network stack. So, you don't need to worry about running on specific ports, etc and also if you write good amount of tests, the time it takes to run all your tests will not be too big, since you deal with in-memory and not network. You should get comparable running times as a typical unit test. Look at this excellent post from Kiran for more details on in-memory testing. In-memory testing will exercise all the components you setup to run in the pipeline but one thing to watch out for is formatters. If you send ObjectContent in the request, there is no need to run media-formatters, since the request is already in deserialized format and hence media formatting does not happen.
If you want to get more closer and willing to take a hit on the running time, you can write your tests using a self-host. Is that what you mean by non-in-memory testing? As an example, you can use OWIN self-hosting. You can use Katana hosting APIs and host your web API and hit it with your requests. Of course, this will use the real HttpListener and the requests do traverse the network stack although it is all happening in the same machine. The tests will be comparatively slower but you get much closer to your prod runs probably.
I personally have not seen anyone using web-hosting and doing lots of integration testing. It is technically possible to fire off your requests using HttpClient and inspect the response and assert stuff but you will not have lot of control over arranging your tests programatically.
My choice is to mix and match, that is, use in-memory tests as much as possible and use Katana-based host only for those specific cases I need to really hit the network.