I have Two Question related to Native App Performance Testing?
1)I have a Payment App, and it comes with bank security which is installed at the time of app installation. It sends an token number and rest of the data in encrypted format. Is it possible to handle such kind of request using Jmeter or any other performance testing tool, do i need to change some setting in app server or jmeter to get this done ?
2)Mobile App uses Device ID, so if i simulated load on cloud server it will use same Device ID which i used while creating script? is it possible to simulate different mobile ID to make it real-time?
any Help or references will be appreciated ..:)
(1) Yes. This is why performance testing tools are built around general purpose programming languages, to allow you (as the tester) to leverage your foundation skills in programming to leverage the appropriate algorithms and libraries to represent the same behavior as the client
(2) This is why performance testing tools allow for parameterization of the sending datastream to the server/application under test
I'm not an expert in JMeter. But work a lot with Loadrunner (LR) (Performance Testing Tool from HP). Though JMeter and LR are different tools, they work under same principle and objective and so objective of performance testing.
As James Pulley mentioned, the performance testing tool may have the capability. But the question is,
Have your tried recording your app with JMeter? Since your app is a native kind, please do the recording from simulator/emulator and check the feasibility. JMeter might not be the right candidate for mobile app load testing.
Alternatively there are lot of other tools available (both commercial and opensource) in market for your objective.
Best Regards
With the raise of several mobile network technologies, load testing a mobile application has become a different ball game in comparison with normal web app load testing. This is because of the differences in the response times that occur in different mobile networks such as 2G, 3G, 4G, etc. Additionally the client being a mobile device has plenty of physical constraints such as limited CPU, RAM, internal storage etc. All of these need to be considered while conducting performance testing of a mobile application if one wants to simulate a scenario close to a real time condition.
Coming to your 2 questions,
1) Yes it is possible but the amount of manual effort that needs to be invested to make the script execution ready might vary (since you are mentioning there is data in encrypted format - some are easy to understand and some are just crude and difficult to handle using JMeter). But there might not be any app server setting that would be required to change (unless of course you are unable to handle the encryption with JMeter in which case, the encryption might have to be disabled for QA phase)
2) As rightly said by James Pulley, these values can be parameterized. However, I fear that these values will be validated by the app server and hence the values need to be appropriately fed in the requests.
You can refer to this link for reference on how to do Mobile Performance Testing for Native application http://www.neotys.com/documents/doc/neoload/latest/en/html/#4234.htm#o4237
.The same could be extrapolated to JMeter to an extent.
Related
Most of the tools out there do not seem to be interpreting jQuery / Javascript code that is loaded on the page that I want to test. This is however important because that code will, in turn initiate a number of calls to other endpoints in my webapp which adds up to load in a real life scenario.
I've looked at JMeter already and am a bit reluctant to give it a try as the landing page mentions explicitly that it does not do Javascript interpreting at all.
What would be some recommendation of tools that can provide a more accurate measurement of load testing by including Javascript code?
None of the load testing tools really executes JavaScript, they all act on protocol level and JavaScript is being executed by browser.
There are 3 options how you could proceed if you want 100% realistic testing:
HP LoadRunner has TrueClient protocol which is basically headless browser with JavaScript capabilities.
JMeter with WebDriver Sampler plugin - the way to kick off real browsers from JMeter test
Selenium Grid (or other way to kick off several Selenium instances at once)
All 3 above options are very resource intensive, you will required at least 1 CPU core and a couple of gigabytes or RAM for a single browser instance so I would recommend a little bit different approach. JMeter cannot execute JavaScript, but it can execute JavaScript-driven HTTP Requests so create main load using JMeter (or equivalent) and while the load test is running use Selenium to automatically check real browsing experience or YSLow to do the same manually.
Load testing tools doesn't execute JavaScript.
You know which endpoints your JavaScript code is using so just add these endpoints to your JMeter scenario.
The thick client problem has been around since the early 1990s with traditional thick client-server applications. In fact, the earliest commercial tools were all driving full clients on X Windows before adding API level virtual user capabilities. On the commercial front this ability to drive full, thick clients is still expressed as GUI virtual users (Mercury/HP/Microfocus LoadRunner, Segue/Borland/Microfocus Silk Performer, Rational/IBM Performance Tester) allowing you to measure the weight between API and user level if needed.
Here is the thing, for a given business process and data set the end client is predictable in its behavior, and in the requests that it makes to the servers that are feeding it data. There may be a few odd conditions where you need to reproduce an algorithm from JavaScript to C (or even run it directly in some tools) for branching on a type of API/HTTP request, but these are not numerous.
How is the community handling performance testing of their secured web areas? We don't particularly have a public facing web site, thus users have to be logged into be able view data / access the system. To further complicate matters, we can not allow users to be logged in multiple times -- if you attempt to login a second time your first session is invalidated. We could turn this feature off (as well as second-level caching), but then we are testing a system which is inherently different from production.
What methodologies should we look into to stress test our application?
Our developers are pretty proficient with Java and Python.
Good question.
Normally we'd use something like Selenium to automate a web-browser talking to the web application itself. This is a system-level approach, and has several advantages:
You are measuring the performance of client-browser too
You can see (to some extent) if the site performs better or worse in different browsers
It is compatible with techniques which do not lend themselves to "raw" web driver programs like ApacheBench
Of course it can take a large amount of work to create automated tests which are representative of real users actions.
Normally you'd have some special test-system with known hardware (ideally similar to production) and a database which includes certain objects which the test suite expects to find. You could also load a production-size (or bigger) simulated data set into this system.
If you used (for example) Selenium to automate functional tests, the functional tests could be reused to build a performance-test suite. That's what we did before.
Anyone can point out the difference between virtual user and real user?
In the context of web load testing, there are a lot of differences. A virtual user is a simulation of human using a browser to perform some actions on a website. One company offers what they call "real browser users", but they, too, are simulations - just at a different layer (browser vs HTTP). I'm going to assume you are using "real users" to refer to humans.
Using humans to conduct a load test has a few advantages, but is fraught with difficulties. The primary advantage is that there are real humans using real browsers - which means that, if they are following the scripts precisely, there is virtually no difference between a simulation and real traffic. The list of difficulties, however, is long: First, it is expensive. The process does not scale well beyond a few dozen users in a limited number of locations. Humans may not follow the script precisely...and you may not be able to tell if they did. The test is likely not perfectly repeatable. It is difficult to collect, integrate and analyze metrics from real browsers. I could go on...
Testing tools which use virtual users to simulate real users do not have any of those disadvantages - as they are engineered for this task. However, depending on the tool, they may not perform a perfect simulation. Most load testing tools work at the HTTP layer - simulating the HTTP messages passed between the browser and server. If the simulation of these messages is perfect, then the server cannot tell the difference between real and simulated users...and thus the test results are more valid. The more complex the application is, particularly in the use of javascript/AJAX, the harder it is to make a perfect simulation. The capabilities of tools in this regard varies widely.
There is a small group of testing tools that actually run real browsers and simulate the user by pushing simulated mouse and keyboard events to the browser. These tools are more likely to simulate the HTTP messages perfectly, but they have their own set of problems. Most are limited to working with only a single browser (i.e. Firefox). It can be hard to get good metrics out of real browsers. This approach is far more scalable better than using humans, but not nearly as scalable as HTTP-layer simulation. For sites that need to test <10k users, though, the web-based solutions using this approach can provide the required capacity.
There is a difference.
Depends on your jmeter testing, if you are doing from a single box, your IO is limited. You cant imitate lets say 10K users with jmeter in single box. You can do small tests with one box. If you use multiple jmeter boxes that s another story.
Also, how about the cookies, do you store cookies while load testing your app? that does make a difference
A virtual user is an automatic emulation of a real users browser and http requests.
Thus the virtual users is designed to simulate a real user. It is also possible to configure virtual users to run through what we think a real users would do, but without all the delay between getting a page and submitting a new one.
This allows us to simulate a much higher load on our server.
The real key differences between virtual user simulations and real users is are the network between the server and thier device as well as the actual actions a real user performs on the website.
I have been tasked with looking for a performance testing solution for one of our Java applications running on a Weblogic server. The requirement is to record production requests (both GET and POST including POST data) and then run these requests in a performance test environment with a copy of the production database.
The reasons for using production requests instead of a test script are:
It is a large application with no existing test scripts so it would be a a large amount of work to write scripts to cover the entire application.
Some performance issues only appear when users do a number of actions in a particular order.
To test using actual user interaction with the system not an estimation at how the users may interact with the system. We all know that users will do things we have not thought of.
I want to be able to fix performance issues and rerun the requests against the fixed code before releasing to production.
I have looked at using JMeters Access Log Sampler with server access logs however the access logs do not contain POST data and the access log sampler only looks at the request URL so it cannot simulate users submitting form data.
I have also looked at using the JMeter HTTP Proxy Server however this can record the actions of only one user and requires the user to configure their browser to use the proxy. This same limitation exist with Tsung and The Grinder.
I have looked at using Wireshark and TCReplay but recording at the packet level is excessive and will not give any useful reports at a request level.
Is there a better way to analyze production performance considering I need to be able to test fixes before releasing to production?
That is going to be a hard ask. I work with Visual Studio Test Edition to load test my applications and we are only able to "estimate" the users activity on the site.
It is possible to look at the logs and gather information on the likelyhood of certain paths through your app. You can then look at the production database to look at the likely values entered in any post requests. From that you will have to make load tests that approach the useage patterns of your production site.
With any current tools I don't think it is possible to record and playback actual user interation.
It is possible to alter your web app so that is records and logs every request and post against session and datetime. This custom logging could be then used to generate load test requests against a test website. This would be some serious code change to your existing site and would likely have performance impacts.
That said, I have worked with web apps that do this level of logging and the ability to analyse the exact series of page posts/requests that caused an error is quite valuable to a developer.
So in summary: It is possible, but I have not heard of any off the shelf tools that do it.
Please check out this Whitepaper by Impetus Technologies on this page.. http://www.impetus.com/plabs/sandstorm.html
Honestly, I'm not sure the task you're being asked to do is even possible, let alone a good idea. Depending on how complex the application's backend is, and how perfect you can recreate the state (ie: all the way down to external SOA services or the time/clock), it may not be possible to make those GET and POST requests reproduce the same behavior.
That said, performance testing against production data is always great, but it usually requires application-specific knowledge that will stress said data. Simply repeating HTTP GETs and POSTs will almost certainly not yield useful results.
Good luck!
I would suggest the following to get the production requests and simulate the accurate workload:
1) Use coremetrics: CoreMetrics provides such solutions using which you can know the application usage patterns. This would help in coming up with an accurate workload model. This model can then be converted into test scripts and executed against a masked copy of production database. This will provide you accurate results about the application performance in realtime.
2) Another option would be creating a small utility using AOP (Aspect oriented apporach) so that it can trace all the requests and corresponding method traces. This would help in identifying the production usage pattern and in turn accurate simulation of workload. AOP frameworks such as AspectJ can be used. This would not require any changes in code. The instrumentation can be done on the fly. The other benefit would be that thi cna only be enabled for a specific time window and then it can be turned off.
Regards,
batterywalam
what is the best tool (open or commercial) currently available, that lets me send customized requests to a web server and get back a response to check the performance?
i will be sending it a load of more than 20K per second, but i need to get numbers for each call made. also, the numbers might be in some microseconds or nanoseconds. How in this small measurement unit, can i work out a baseline and a benchmark?
If you're using Apache, Apache AB is a benchmarking to test how many requests your serve can serve per second and how well it handles load and concurrency. It's an open-source project - check it out here.
In addition, wikipedia has a nice list of benchmarking software for testing servers.
You can use the Web Application Stress Tool of Microsoft
The Microsoft WAS web stress tool is designed to realistically simulate multiple browsers requesting pages from a web site. You can use this tool to gather performance and stability information about your web application. This tool simulates a large number of requests with a relatively small number of client machines. The goal is to create an environment that is as close to production as possible so that you can find and eliminate problems in the web application prior to deployment.
You can find a list of Open Source software for performance (most of them are for web that send custom request to webserver).
Don't know if either of these have granularity better than milliseconds but check out JMeter (open source) and LoadRunner (Commercial). LoadRunner is not cheap but it allows you to span load generation across multiple machines with aggregated results.