I would like to load test pages AND render javascript. It seems there are 3 categories of applications which might solve this problem, and all three miss the mark.
1) Jmeter, apachebench, tsung, Grinder, Iago
Why it wont work: It doesn't render javascript. These are handy tools, but they won't work for this purpose.
2) Watir, Selenium
These tools are excellent, they use real browsers and render javascript / ajax, but alas, they are designed mostly for functional testing, and not performance testing. You could create your own app to use these things for a load test, but it would be a massive pain in the ass to collect all of the performance metrics and aggregate them.
If only there was a combination of the first two types of web performance tests, that would be great.
The third option, solves this problem, but unfortunately you have to pay for it.
3) Web load testing services
Services like Keynote Load Pro, BrowserMob and others are great, and they solve the problem of using real browsers and rendering JavaScript. The only problem is, I don't want to pay $300 ever time I run a damn load test (this is an exaggeration, but not really, depending on how many virtual users you use).
So that option won't work, unless I want to hemorrhage money.
Isn't there a test harness out there that solves this problem? It seems like a big gaping hole where there is a need for an open source tool that no one has solved yet. The commercial companies dominate this space (and those who want to employ a full time developer to write a selenium performance test framework).
The good thing about jmeter, ab, tsung, etc. is they can give you aggregated metrics in relatively short time but they can't execute javascript. They can't measure how fast DOM has been created. They won't load external resources such as images, javascripts, styles. They are good for stress-testing which is not the case here.
When measuring web performance you need to know:
what do you really want to measure? - domLoading, domInteractive, domContentLoaded or maybe something else from Performance Timing Interface,
which statistical measure you want to use? - p95, median. Average is not the best choice.
when? - do it always at the same time to make tests comparable,
from which location? - the best you can do is to spawn dedicated server located closely to your production,
how confident you want to be about the results? - choose sample data size, for example if you accept 3% margin of error with 95% confidence interval then choose 1067.
For doing such Synthetic Performance Measurements you can use https://github.com/msn0/sweter. It measures timeToFirstByte, domInteractive and domComplete timings. It can be scheduled to run tests always at the same time. Sweter can report raw data to ElasticSearch, console, json file or wherever you want.
The different topic is Real User Measurements. There are good tools for that already, e.g. NewRelic.
Well you haven't mentioned what metrics you really want to measure. But I think you can test both server side and browser performance using two simple tools.
PhantomJS and HAR file: PhantomJS is a webkit headless browser that has a Javascript API. You can script it using Javascript to do pretty much everything (including javascript rendering). Use it to generate HAR file so that you can analyze performance bottle necks in all the requests.
Navigation Timing API : Modern browsers support navigation timing data which gives you details about timings in browser rendering, including all important metrics like domLoading, domInteractive etc.
Related
What are the best tools to test the performance of a (not deployed) application using Play framework? Things like, how long takes a request to execute, with different parameters, simulating a lot of requests (stress test), etc.
I'm searching a while but the problem is that the keyword "performance", "benchmarks" etc. lead me to pages about the performance of Play framework.
I thought maybe functional tests, could be used to measure performance (print difference between method start time and end...). But this doesn't look suitable for this kind of task.
I could just write a script, that triggers the requests, writes the timestamps to a log file... but maybe there's something finished, with extras, like e.g. charts, etc.
Any hint in the right direction greatly appreciated.
Iago is a load generation tool by Twitter written in Scala. Also, I've used the Loader.io addon on Heroku to do performance testing. Loader.io also has a non-heroku service that I have not used. Iago is probably your best bet for local testing of a non-public app.
A good example is a project used by Versal to choose their Scala stack for production.
The project is Scamper.
I am not able to find out anywhere that how can we do performance test manually.
Please help me out for this query.
Thanks!
Maybe you are looking for JMeter or a similar tool.
What browser? Most of the current browsers support the W3C Navigation Timing spec and expose performance data directly on the DOM. You can access it from the console, from javascript on your pages or from browser extensions that display the information.
If you want more detail like a resource load waterfall then you can usually access that directly from the dev tools provided by the various browsers.
One thing you will want to be really careful of is to make sure you do your testing in a configuration that is similar to the users. If you are running a server locally and testing from a browser on the same machine or even the same network then your performance data will be pretty worthless (unless it's an intranet app).
you can perform manual testing (Performance testing) for any webpage by optimizing your css, Javascript and images ( size).
I think JMeter is a best tool for same to check webpage testing if you want add some scripting you can also add.
Also you can check Yslow addons of firefox.This addons give you filter data to optimized your page perfromes.
Also there are some online link available.
How can we run performance testing manually for any webpage?
You can simple use GTMatrix tool which will response of your site Performaces overall in detail.
The best way to go for Performance Testing without any tool is to provide a Standard loading time for each page as per one's experience knowledge. Else request the client to provide an ideal time for each page. Against which the loading time can be verified. But in case of multiple user simultaneously JMeter is the best hands on Approach available. Its Open source. Easy to understand. And you get reports too.
But of course there are multiple factors that would hinder the Performance. They are :
Your network speed
The Server speed on which your application is hosted
The number of Simultaneous users using
The Heavy images in pages
Last but not the least unnecessary links, codes, in short memory consumption in Code, could be loops not required. All the gifts from Developer Teams !!
There are already plenty of hosted services for tracking server response times but we're looking for something to track page render/load times (i.e. browser renderer).
The problem with PageSpeed, YSlow, etc is they're on request where we want something that runs constantly and takes a reading every 15 mins for example.
Latest browsers have a window.performance.timing property which contains the timestamp at which some events occurred (such as domainLookupStart, domLoading, domInteractive, ...).
You may want to send a sample of those numbers to your servers.
See https://developer.mozilla.org/en/API/navigationTiming
Google Analytics has a page speed metric, you may want to look at it too.
Atatus monitors page load time and there by does Real User Monitoring (RUM), and also it monitors for JavaScript errors along with capturing all the user actions that lead to the error.
It also gives various views of how your performance is across geographical locations and browsers.
https://www.atatus.com/
https://www.atatus.com/blog/announcing-real-user-monitoring/
Disclaimer: I’m a web developer at Atatus.
We use GTmetrix to get daily Page Speed and YSlow scores. If you need a higher frequency, you can automate YSlow or Page Speed using Selenium or ShowSlow.
Try a free tool like dynaTrace's AJAX Edition 3. From the website: "Speed your page load times, optimize rendering, tune DOM execution, and compare to competition. Even integrate with Selenium, Watir or QTP to begin automating your performance tests. It's free, it's easy and it's now for both Firefox and IE."
I do work for dynaTrace, in full disclosure. But this simple, free tool should be very useful to you.
Best
GP
This isn't a question of what stress testing tools are out there. I'm afraid it's a lot harder than that. (At least for me)
Consider a restful architecture for a forum or blog that generates random IDs for each post.
Simulating creating those topics/articles would be simple, because you'd just be posting form data to an endpoint like: /article, or /topic
But how do you then stress test commenting on those articles/topics? This is different, because the comments need to belong to an article/topic, which means that you need the ids of those items. However, if all you can do is issue posts, and you have no way of pulling those ids, you'd be unable to create them.
I'm creating a site that is similar in this regard, and I have no idea how to stress test the creation of the comments.
I have two ideas, and they're both pretty awful:
Generate a massive system ahead of time with some kind of factory, and then freeze it. From there, I figure I'd have to use some kind of browser automation to create my 'comments' on all of this. The automation would I suppose go through a recording proxy, like what JMeter offers. Then, to run the test, I reload the database, and replay the massive log file.
Use browser automation for the whole thing, taking advantage of the dynamic links delivered in the HTML page. The only option here would be Selenium, and really, we're talking a massive selenium grid that would be extremely expensive. Probably very difficult to maintain also.
Option 2 is completely infeasible near as I can tell, but option 1 sounds excruciating. I'm really hoping someone can suggest something more clever.
Option 1.
I mean, implementation notes aside, you're basically just asking for a testing environment. So, the answer is to make one. In whatever fashion:
Generate it
Make it once and reload it
Randomise it
Whatever. It's the approach to go with.
How do you your testing is kind of a side issue (unit testing/browser/whatever, up to you).
But you've reached a point where you need to test with real data. So make it happen.
This is a common problem. We handle it by extracting the dynamic parts of the URLs from the server responses. I presume this system uses web browser client - which implies that those URLs are being sent in the server responses. If they are in the responses, then you CAN get them. However, since you said "if all you can do is issue posts, and you have no way of pulling those ids", then perhaps this is not the case? In that case, can you clarify?
We've recently been doing a lot of testing of Drupal systems for our customers - which has exactly the problem you've described. We either solve it by extracting the IDs dynamically from the page as the user browses to the page they want to comment on, or we use option 1, or a combination of both. Note that if you have a load testing tool handy, then generation of content is not too difficult - use the tool to do it. I.e. run a "content generation" load test. Besides yielding useful data on its own accord, that will give you a test database that you can then backup/restore as needed to maintain your test infrastructure. Now you can run the test on a more realistic environment - one that has lots of content already in it (assuming, of course, that this is realistic for your purposes).
If you are interested, I'd be happy to demo how we solve the problem using our software (Web Performance Load Tester).
I have used Visual Studio to solve this kind of problem. Visual Studio allows C# coded web tests that can programatically parse the html returned and take action based on that.
I was load testing a SharePoint website and required information to be populated ahead of time. I did create a load test that was specifically for creating "random" pages of content ahead of time. I populated a test harness database with the urls ahead of time, allowing some control over the pages that were loaded.
With a list of "articles" available and a list of potential comments, it is possible to code a pseudo-random number generator (inside a stored procedure because of the asynchronous nature of the test harness) to get a repeatable load test. That meant that the site would be populated in the same way each time the load test was run.
It does take some effort to create a decent way of populating the site off the bat, but the return in the relevance of the load test is quite good.
Are there any open source (or I guess commercial) packages that you can plug into your site for monitoring purposes? I'd like something that we can hook up to our ASP.NET site and use to provide reporting on things like:
performance over time
current load
page traffic
SQL performance
PU time monitoring
Ideally in c# :)
With some sexy graphs.
Edit: I'd also be happy with a package that I can feed statistics and views of data to, and it would analyse trends, spot abnormal behaviour (e.g. "no one has logged in for the last hour. is this Ok?", "high traffic levels detected", "low number of API calls detected") and generally be very useful indeed. Does such a thing exist?
At my last office we had a big screen which showed us loads and loads of performance counters over a couple of time ranges, and we could spot weird stuff happening, but the data was not stored and there was no way to report on it. Its a package for doing this that I'm after.
It should be noted that google analytics is not an accurate representation of web site usage. This is because the web beacon (web bug) used on the page does not always load for these reasons:
Google analytics servers are called by millions of pages every second and can not always process the requests in a timely fashion.
Users often browse away from a page before the full page has loaded and thus there is not enough time to load Googles web beacon to record a hit.
Google analytics require javascript to be installed which can be disabled.
Quite a few (but not substantial amount) of people block google-analytics.com from their browsers, myself included.
The physical log files are the best 'real' representation of site usage as they record every request. Alternatively there are far better 'professional' packages, of which Omniture is my favourite, which have much better response times, alternative methods for recording actions and more functionality.
If you're after things like server data, would RRDTool be something you're after?
It's not really a webserver type stats program though, I have no idea how it would scale.
Edit:
I've also just found Splunk Swarm, if you're interested in something that looks "cool".
Google Analytics is free (up to 50,000 hits per month I think) and is easy to setup with just a little javascript snippet to insert into your header or footer and has great detailed reports, with some very nice graphs.
Google Analytics is quick to set up and provides more sexy graphs than you can shake a stick at.
http://www.google.com/analytics/
Not Invented here but it's on my todo list to setup.
http://awstats.sourceforge.net/
#Ian
Looks like they've raised the limit. Not very surprising, it is google after all ;)
This free version is limited to 5 million pageviews a month - however, users with an active Google AdWords account are given unlimited pageview tracking.
http://www.google.com/support/googleanalytics/bin/answer.py?hl=en&answer=55543
http://www.serverdensity.com/
One option is to use external monitoring tools, which will monitor the web performance from outside the firewall by simulating end user activities.
Catchpoint Systems has an interesting approach that requires very little coding and gives you the performance stats from outside the datacenter and from inside the asp.net (like processing time, etc)
http://www.catchpoint.com/products.html