Suppose I want to test a server's and its web application's ability to handle many simultaneous connections well and show decent latency.
So ideally I would want a thousand machines to bombard it with usage requests, but that's not practicable. So instead, can I just make a testing script with a thousand threads to run on the same server and have them perform the testing, connecting to the server via a geographically far-away proxy?
My reasoning here is that the signal will have to travel realistically big distances to the proxy and back, so that sort of emulates the reality of real clients accessing the server.
Then again, to take this one step further, are there prepackaged emulators/frameworks that could perform a similar test without using internet at all, just simulating the latency of the network, realistically creating all the socket connections and other resource intensive stuff etc?
There are lots of load testing products (open source and commercial) that have bandwidth/latency simulation. Running through various remote proxies is overly complex. On the open source side, look at JMeter and Pylot. On the commercial side, my company, BrowserMob, provides a website load testing product that uses the Amazon cloud to generate load. We actually do generate load from thousands of machines/IPs, without breaking the bank :)
Related
I'm currently running some performance tests to see how many requests per second a newly developed web back-end can handle.
However, I have absolutely no idea how many requests per second I should expect the web server to handle (10? 100? 1000?).
I'm currently testing on a modest 1GB - 1 core virtual machine. What should be a reasonable minimum number of request/second such a server should be able to handle?
I think the right question you should be asking yourself is what performance goals I want my application to have when X requests are being handled?
Remember that a good performance test is always oriented in achieving realistic and well defined performance goals.
These goals are usually set by the performance team and the customers/stake holders.
There are many variables to this question;
What web server software are you using (Apache, nginx, IIS, lighttpd, etc)? This affects the lookup latency and how many simultaneous requests can be handled.
What language is your system logic written in (PHP, Ruby, C, etc)? Affects memory usage and base speed of execution.
Does your system rely on any external services (databases, remote services, message queues, etc)? I/O latency.
How is your server connected to the outside world (dedicated line, dial-up modem (!), etc.)? Network latency.
One way to approach this is to first discover how many requests your webserver can serve up in optimal conditions, eg. serving a single static HTML page of 1 byte with minimal HTTP headers. This will test the web server's fundamental receive-retrieve-serve cycle and give you a good idea of it's maximum throughput (handled requests per second).
Once you have this figure, serve up your web application and benchmark again. The difference in requests per second gives you a general idea of how optimal (or sub-optimal) your app is.
Even the most modest of hardware can deliver thousands of responses given the right conditions.
I am looking to load test a consumer website. I have tried using JMeter. However, in that case, all the requests originate from one machine. What I really want is to simulate real users across the country some on low speed dialup connections and others on highspeed.
What are the best practices to follow in such a scenario?
JMeter supports distributed testing - so if you're already comfortable with it as a tool, you can use it to power these distributed requests from an arbitrary number of machines, too.
Note that all machines run the exact same test plan, so either your plan should have some random selection of fast/slow environments, or you may be able to select which profile to use based on some system properties.
You might want to consider using a 3rd party service such as Load Impact.
You've expressed two different but related concerns - traffic coming from a single machine and simulating various end-user network speeds.
Why is the first one a concern for your testing? Unless you have a load balancer that uses the IP address as part of its load-distributing algorithm, the vast majority of servers (and application platforms) don't care that all the traffic is coming from a single machine (or IP address). Note also that you can configure the OS of your load generator for multiple IP addresses and the better load-testing tools will make use of those IP addresses so that traffic comes from all of them.
For the simulation of end-user network speeds, again, the better load-testing tools will do this for you. That can give you a pretty good feel for how the bandwidth will affect the page load time, without actually using distributed load generation. But tools frequently do not account for latency. That is where there is no substitute for distributing your load generation.
You can do distributed testing with JMeter, though it can be a bit cumbersome. How many locations do you need? Without knowing more about what you need, my first suggestion would be to choose a tool that has features designed specifically to do what you need. I will pimp our product, Web Performance Load Tester, but there are certainly other options. Load Tester can emulate various end-user connection speeds and has built-in support for generating load from Amazon EC2 (US east and west coast and Dublin, IR...support for Asia coming soon). Once you set up an EC2 account, you can be running your first test from the cloud in 10 minutes.
I recently created a turn-based game server that can accept 10s of thousands of simultaneous client connections (long story short - epoll on Linux). Communication is based on a simple, custom, line-based protocol. This server allows clients to connect, seek for other players in game matches, play said games (send moves, chat messages, etc.), and be notified when the game has ended.
What I'm looking to do now is test the server by simulating client connections. I'm hoping to support 10s of thousands of simultaneous connections, so this testing is very important to me. What do you guys use for your own testing?
Some things I'm researching now are: pexpect (python expect lib for the functional testing) and tsung for load testing.
I'd like to be able to just test from my laptop since I do not have a cluster of client machines to connect from. Perhaps I'd need to use ip aliasing or some-such in order to generate 100s of thousands of outbound sockets (limit is 65K per interface AFAIK).
Anyway, it seems to me like I need something fairly custom but I thought I'd ask before I went down that path.
Thanks!
I've used JMeter with custom sampler and assertion components before to do automated regression/load testing for a banking application with a custom protocol (Java RMI based API).
It's not exactly lightweight though, and you'll end up doing a lot of extra coding in the JMeter components to support your custom protocol. I'm guessing you'd have to code your own Java socket based client in this case.
But it gives you a lot of flexibility in defining the logic for testing the components, so you can do whatever you want inside there. It scales nicely as well, and allows you to throw a lot of concurrent connections at the system under test.
I decided it was best to "roll my own" to start with.
We are using HP LoadRunner it's the state of the art load testing product. (But also an expensive one). It can simulate thousands of requests to the server and provides metrics on response time etc..
what is the best tool (open or commercial) currently available, that lets me send customized requests to a web server and get back a response to check the performance?
i will be sending it a load of more than 20K per second, but i need to get numbers for each call made. also, the numbers might be in some microseconds or nanoseconds. How in this small measurement unit, can i work out a baseline and a benchmark?
If you're using Apache, Apache AB is a benchmarking to test how many requests your serve can serve per second and how well it handles load and concurrency. It's an open-source project - check it out here.
In addition, wikipedia has a nice list of benchmarking software for testing servers.
You can use the Web Application Stress Tool of Microsoft
The Microsoft WAS web stress tool is designed to realistically simulate multiple browsers requesting pages from a web site. You can use this tool to gather performance and stability information about your web application. This tool simulates a large number of requests with a relatively small number of client machines. The goal is to create an environment that is as close to production as possible so that you can find and eliminate problems in the web application prior to deployment.
You can find a list of Open Source software for performance (most of them are for web that send custom request to webserver).
Don't know if either of these have granularity better than milliseconds but check out JMeter (open source) and LoadRunner (Commercial). LoadRunner is not cheap but it allows you to span load generation across multiple machines with aggregated results.
Can you suggest how to create a test environment to simulate various types of bandwidths and traffic in a web app?
Or maybe an open source program which does this against localhost?
I think this is a very important subject when programming web apps but it is not a usual topic, the only way i can imagine to create such kind of environment is to use some kind of proxy in a local network but before start looking into the squid documentation i would like to hear your suggestions.
if you're using apache you may want to take a look at apache ab
There are two approaches to shape network traffic to simulate a network link:
Run some software on the client or server that sits somewhere in the networking stack and shapes the traffic between the app and the network interface
Run the traffic shaping software on a dedicated machine with 2 network interfaces through which your traffic is routed
(2) is a better solution if you don't want to install software on the client or server (and possibly impact performance), but requires more hardware fiddling.
Some other features you might want to think about are what shaping parameters can be simulated. Most do delay and packet loss, some do jitter and bandwidth limiting as well. Some solutions can selectively filter traffic (for instance by port number, TCP or UDP etc).
Here is a list of some of the systems I've found:
Open Source or Freeware
DummyNet is an open source BSD Unix-based for dedicated devices. It is not clear if the software is being actively maintained
NistNet is an open source Linux-based system for dedicated devices. The software has not been actively maintained for several years.
Commercial
Apposite Technoligies sell dedicated hardware solutions for simulating WAN links, with a Web based GUI for configuring the settings and collecting traffic measurements
East Coast DataCom sell hardware dedicated simulators for simulating routers and modems
Itrinegy offer both dedicated device solutions, and solutions for running on clients or servers.
Network FX offer several dedicated device products for simulating network impairments between the client & server
NetLimiter is a client side system that allows throttling of individual applications, and includes a firewall.
Shunra Software offer a range of products, from high end enterprise WAN simulation and testing, to a simple client-resident emulator.
The closest I can think of is doing something similar with VEDekstop from Shunra..
Simulating High Latency and Low Bandwidth in Testing of Database Applications
Shunra VE Desktop Standard is a Windows-based client software solution that simulates a wide area network link so that you can test applications under a variety of current and potential network conditions – directly from your desktop.
I wrote a php script awhile back which used CURL to run a sequence of page requests against my server which represented a typical use scenario. I had it output the times that it took for the server to respond to each of the requests. I then had another script which spawned a bunch of these test case scripts simultaneously for a sustained period and correlated the results into a file which I could then look at in a spreadsheet to see average times. This way I could simulate the number of users hitting the site that I wanted. The limitations are that you need to run the test script on a different server to the web server and that the client machine can become too loaded to give meaningful results past a certain point. I've since left the job otherwise I would paste the scripts here.
If you are running a Linux box as your server, Linux box as your client, or have the capability to put (perhaps a VM) a Linux router between your client and server, you can use NetEm.
NetEm is a Linux TC (Traffic Control) discipline which can delay (i.e. add latency) packets leaving a host. Although it's tricky to set up clever rules (e.g. add latency to some traffic, not to others), it's easy to add a simple "delay everything leaving the interface by 50ms" type rules and some recipes are provided.
By sticking a Linux VM between your client and server, you can simulate as much latency as you like. And you can turn it on and off dynamically. Linux has other TC disciplines which can be combined with NetEm to restrict bandwidth (but the script to set this up can be somewhat complicated). NetEm can also randomly drop packets.
I use it and it works a treat :)
Web Application Stress Tool (WAST) from Microsoft is what you need.
http://www.microsoft.com/downloads/details.aspx?familyid=e2c0585a-062a-439e-a67d-75a89aa36495&displaylang=en
I haven't used it for years (lack of need, not because I'd found anything else), but xat webspeed would be the first thing I would point toward
As other people have mentioned, Apache's ab (comes with Apache, so you probably have it already) is good.
Other good options are:
HP's LoadRunner Apache
Jakarta's JMeter
Tsung (if you want to get your erlang on)
I personally like ab and JMeter the best.
We use Loadrunner to do bandwidth and traffic simulation in our App. Loadrunner is can start agents on various machines and you can simulate one machine as running on dialup modem v/s another on DSL v/s another on Cable internet.
We also use Loadrunner to simulate various kinds of traffic conditions from 10 user run to 500 user run. We can also insert think times in the script and simulate a real user executing the http request. The best part is that it comes with a recording studio where it will plug in with Internet explorer and you can record the whole scenario/Usecase that can be as simple as hitting one page to a full blown 50-60 page script or more.
i found this little java program that works great : sloppy
yet not a proffesional solution but it works for simple tests, i guess it uses java streams and buffers to slow down the connection .
Have you looked at Tsung? It's a great utility for seeing if your website will scale in event of attack, I mean massive popularity. We use it for our web frontend, and our internal systems too.
If you're interested in performing your tests out of your browser, there is also a really great Firefox plug-in.
Do not forget about Wanulator (http://www.wanulator.de/).
The name Wanulator comes from "WAN" and "simulator. This pretty much describes what the software does: It simulates different Internet conditions such as delay or packet loss. Furthermore it simulates user access line speeds e.g. modem, ISDN or ADSL.
Wanulator is currently packaged as a Linux boot CD based on SLAX. This will give you a full out of the box experience. You can turn any PC into a test-system within a blink - just by booting the Wanulator CD. The package already includes useful client SW such as web-browser and network sniffer (Wireshark). Nevertheless if the PC has 2 network interfaces the system can run as an intermediate system between your server and your client - as a switch - without any configuration hassles.