How do you test your caching layer? (be it Varnish, Nginx/Apache, Fastly or any other product/service). Given the fact that caching is hard I've found surprisingly small amount of information on the topic on Internet. Currently I'm using PhpUnit to write 'unit' tests that follow certain scenarios:
set up initial parameters like HTTP headers, cookies, HTTP referrer, user agent,
execute (possibly multiple) HTTP requests,
check those requests for expected result (HTTP headers, cookies, presence or absence of certain HTML code),
check for expected response times,
verify that state of client doing the HTTP request is in sync with application (testing session object contents, database etc.)
I constantly add new tests to my test case and test caching layer on each change.
I've seen people using varnishtest to test Varnish but it seems to be greatly limited in what it can do (for example you can't test backend database to confirm that some state has been set)?
What's your strategy to test caching layer?
I use tsung, that is originally a distributed load testing tool but it can do what you listed.
Tsung are piloted by xml files that describe scenario where you can set initial parameters, launch multiple clients in parallel and check for specific results.
Related
I'm trying to load test a nextcloud server with jmeter. I have set two http request with a cookie manager in jmeter and sometimes it succeed, sometimes not. TOtally randomly.
The first request is a get request on login page setting some cookies which are implemented each time.
The second request is a post request with a redirect and the answer to the post request is supposed to set cookies which are sometimes set sometimes not. When they are set the request succeed, otherwise not.
I have tried all the possible options in the cookie manager but it remains the same. I also have put CookieManager.check.cookies=false in user.properties...
Any help is very welcome
JMeter doesn't do anything "randomly", most probably your application gets overloaded and cannot respond with appropriate Set-Cookie header
Try temporarily enabling saving response data and headers in JMeter and check application responses for the requests which are just before the failing ones.
Another possible reason could be JMeter malfunction, it may happen i.e. if you don't follow JMeter Best Practices or JMeter doesn't have enough headroom to operate in terms of CPU, RAM, Network, etc. Make sure that JMeter is properly configured and has sufficient amount of resources from the underlying operating system. If you don't have any better monitoring toolchain you can consider using JMeter PerfMon Plugin for this.
I recorded login and logoff requests using blazemeter. After the record nearly 10 request has been created by blazemeter, which some of them includes .../signalr/.../connectionToken labels.
when i run the test these labels return an error like and .
the test included 10 users. The users have different username and passwords. The other labels (another from these signalr labels) return true.
So, i wonder now if i can disable these pages and not include in the tests? or
any solution for this issue?
if i can disable these pages and not include in the tests
theoretically yes, you can ask around whether SignalR is in scope for performance testing and if not you could disable those, however I believe well-behaved JMeter test should act exactly like a real browser so if the real browser sends these requests to .../signalr/.../connectionToken - JMeter should also send them
or any solution for this issue
I don't think you can record and successfully replay anything in Web 2.0 times, the majority of web applications heavily use dynamic tokens for various reasons (saving client state, security, tracking, etc.). When you record the request you get a hard-coded value which can expire or invalidate by other means, you need to identify all the dynamic values like this SignalR token and perform correlation: the process of dynamic values extraction from the previous responses using suitable JMeter Post-Processors and re-using them in next requests in form of JMeter Variables or Functions
If you record the same user scenario second time and compare two resulting test plans all the request parameters which change will require correlation. There is also an alternative recording option which is capable of exporting recorded requests in "SmartJMX" mode with automatic detection and correlation of dynamic parameters, see How to Cut Your JMeter Scripting Time by 80% for more details.
I am load testing a scenario in JMeter. I'm trying to achieve a certain amount of r/s and then sustain that for x amount of time. However, with the testing tool, it looks like falling short of r/s for each endpoint. Do I just need more virtual users from the system? or is an additional fix needed for the test plan below?
HTTP Header Manager
User Defined Variables
DNS Cache Manager
HTTP Authorization Manager
HTTP Cache Manager
Thread Group
HTTP Cookie Manager
CSV Data Set Config # 1
HTTP Request #1
HTTP Header Manager
HTTP Request #2
HTTP Header Manager
JSON Extractor
HTTP Request #3
HTTP Header Manager
CSV Data Set Config # 2
HTTP Request #4
JSR2223 PreProcessor
HTTP Header Manager
The CSV Data Set Config # 1 is CSV of username and passwords that are passed to HTTP Requests #1 and #2. The JSON Extractor extracts values from HTTP Request #2 to the HTTP Header Manager of HTTP Request #3.
I can think of 2 situations:
JMeter isn't capable of sending requests fast enough. If this is the case:
Make sure to follow JMeter Best Practices
Try increasing the number of threads/ virtual users
If you cannot conduct the required load from a single load generator - consider switching to Distributed Testing
Your application under test isn't capable of responding fast enough. The reasons could be:
Your application lacks resources (CPU, RAM, Network, Disk, Swap, etc.). Consider monitoring its baseline health metrics using i.e. JMeter PerfMon Plugin
Your application middleware configuration isn't suitable for high loads. Inspect configuration of load balancer, application server, database, whatever to see if they're set up good enough for the anticipated load. Check out Web Performance Optimization tips to get initial idea. Normally the vendors of application servers, databases, etc. provide scaling instructions.
Your application code is not efficient enough, i.e. uses bad algorithms. You can use profiling tools to check what are the slowest and/or frequently called functions, largest objects, etc.
I am using jMeter to load test logging in and logging out of my application. I have a CSV with 500 uname and pwords, and I have also set the active threads to 500.
Is this the correct way to stimulate a load of 500 users?
From technical perspective - yes, JMeter will pick up credentials from CSV and send them along with the request.
However if you think about what happens in reality logging in and immediately logging out doesn't seem a realistic use case for me. The whole idea of load testing a web application is simulating real users as close as possible, it includes:
Application usage scenario(s): how many users will be accessing your application at the same time and what users will be doing what things? For example given Stack OverFlow the vast majority of users are searching for answers by given criteria, some users are typing questions, some of them providing answers, and some of them are commenting. These all are different behavioral patterns which need to be accurately simulated by your load test
Each user should have a "user session" which identifies him so application would distinguish different users so make sure you have HTTP Cookie Manager in your Test Plan
Real users use real browsers which are downloading images, CSS and JavaScript files using parallel thread pool (around 5 concurrent threads) to speed up page loading process. In order to mimic this behavior you need to configure HTTP Request samplers to download embedded resources and do it in parallel. You can apply this configuration to all the HTTP Request samplers via HTTP Request Defaults.
Don't forget to add HTTP Cache Manager to mimic browser cache as real browsers normally cache embedded resources to disk and don't re-request them on subsequent calls.
So I would recommend reviewing your test scenario and/or load pattern and get familiarized with How to make JMeter behave more like a real browser article
I would like to measure loading time of a document in my testing web app. I have used JMeter for this, but I am getting different values for each run. I am measuring average time in the summary report.
I am not sure, that the value is proper or not.Is this approach is correct or Is there any plugin JMeter available?
I have used HTTP watch to get rendering time, but I can't use that tool for more than 1 user (Load Testing). I am using JMeter 2.13. Could you please help me in this?
With the help of aggregate report or csv / xml results you get required information regarding response times BUT
In Jmeter, Response time = Processing time + Latency(time taken by network while transferring data)
In Browser, Response time = Processing time + Latency + Rendering time
Hence you will found a difference between http watch response times and jmeter response times.
If you need to include rendering times also in your response times, then use tools, like loadrunner (commercial), selenium (open source) and so on. Personally in my opinion client side rendering is not a measurable value, unless all of the users accessing the application are having same configuration of hardware, software and network access. However, while JMeter test running with peak load to the system, manually browse the site using various browsers and with the help of developer tools you can find rendering times.
I am getting different values for each run - this will depends upon test data you are using, server health status, network delays and so on.
I doubt you'll be able to get 2 fully identical test run results, there always will be some form of fluctuation caused by underlying hardware and software implementations. You should be receiving similar results with some statistical noise.
If this is not the case, your JMeter test might be misconfigured. From "realness" perspective mind the following configuration:
Make sure you have Retrieve All Embedded Resources from HTML Files box checked and you Use concurrent pool. The easiest way to configure it for all the samplers is using HTTP Request Defaults
Add HTTP Cache Manager to your test plan. Previous setting "tells" JMeter to fetch embedded resources like scripts, styles, etc. from the pages. Real browsers do it as well but they do it only once, on subsequent requests these resources are being returned from browser's cache.
Add HTTP Cookie Manager to your test plan. It represents browser cookies, enables cookie-based authentication and maintains sessions.
Add HTTP Header Manager to represent browser headers like User-Agent, Content-Type, encoding, etc.
When you use a straight HTTP Protocol layer virtual user, independent of the tool (Jmeter, LoadRunner, SOASTA, Grinder, ...) then what you will be timing will be the request/response information coming from the server with very low coloration from the local processing on the client for JavaScript and the final "drawing on the screen" which is rendering.
Up until the point where the server is degraded due to number of requests or network limitations the only area where you can tune is in the page architecture, which if you are waiting to the last 100 yards before deployment to address then you are likely in trouble.
Steve Souders has written quite a bit on the subject of page architecture in his books "High Performance Websites" and related works. In short, the rule of thumb comes down to making fewer requests, smaller responses and serving the data from the closest possible location to the client. These have the effect of minimizing the most expensive finite resource to a web client, the network. For instance, a browser sprite reduces the number of calls for images, minification and compression reduce the size of the transmission and a CDN changes the number of hops to the requested item to a location closer to the end client.
In order to affect changes to page architecture you need to move upstream into your development cycle and your functional testing cycle. You will need to work with development to implement hard gates where code/pages cannot be submitted to the project without first passing performance gates related to design. Your development team and functional testing members will need to respect those gates. As to what the gates should be, I refer you back to the works of Mr Souders as a great source of data for construction of your gate rules.
This gets you to the level of "works for one: Performant for one." Then you can use that as a known good to answer the questions related to server scalability and at which point the service to the client from requests begins to degrade. If you have a CDN in your organization, be sure to take that into account in your test model, for if you do not then you will overload your server vs production.
As far as actual speeding of the "rendering" or drawing on the screen? You need to purchase a faster video card barring changes from the browser manufacturer. Speeding up JavaScript? Make sure that all of your JavaScript is as small and as lean as possible. Have your functional test team test on very dirty browsers with lots of add-ins as well as lower powered hardware for a view of maximum out of spec response. If you need a view of what your standard hardware model looks like from your clients (Browser/OS/some hardware into) then you can process the data in your HTTP request logs and specifically the user agent related to client configuration information.