Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
As a web developer I've been asked (a couple of times in my career) about the performance of sites that we've built.
Sometimes you'll get semi-vague questions like "will the site continue perform well, even during product launch week?", "can the site handle a million users?", and even "how is the site doing?"
Of course, these questions are very legitimate, and I have always tried to answer these questions to the best of my ability, using a combination of
historic data (google analytics / IIS logs)
web load test tools
server performance counters
experience
gut feeling
common sense
a little help from our sysadmins
my personal understanding of the software architecture in question
I have usually been able to come up with reasonable answers to these questions.
However, web app performance can be influenced by many things (database dependencies, caching strategies, concurrency issues, etcetera, user behaviour).
I'm a programmer and not a statician, and my approach to this problem has always felt deeply unscientific. So I did a little more research... and all of my google results seem to focus on tools and features and metrics (and MORE metrics) when I am really looking for a way to make sense of these things.
The question:
What are some good resources (books?) to read on the best practices for a developer to read on the subject of web load testing, that will help me answer these types of questions?
First your question proves you do understand the problem. It can sometimes be tricky enough creating the tools, scripts etc. to generate the load but the real challenge lies in evaluating the results and what to monitor.
A very easy answer to your question could be to Generate load on a production-like environment that is similar to current or expected usage. If it runs ok without any crashes or slow performance that is usually good enough. After that, increase load to see where your limits are.
When you reach your limit my experience is that this is purely a project budget question. Will we invest more time/money/resources etc to evaluate the cause.
I work as a test professional and I do recommend respect load testing as a vital part of the development process but unfortunately that is not always in line of what management decides.
So the answer to your question is that almost everyone needs to be involved in this process:
developers to monitor their code; system admins need to monitor CPU, memory usage etc.; DBA; networking guys; and so on. They all probably need their own source of knowledge to be able to get all this info recorded and analysed.
A few book tips:
The Art of Application Performance Testing: Help for Programmers and Quality Assurance
http://www.amazon.com/exec/obidos/ASIN/0596520662/
The Art of Capacity Planning: Scaling Web Resources
http://www.amazon.com/exec/obidos/ASIN/0596518579/
Performance Testing Guidance for Web Applications
http://www.amazon.com/exec/obidos/ASIN/0735625700/
Have you seen:
Performance Testing Guidance for Web Applications by J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea
It's even available on the web for free.
http://msdn.microsoft.com/en-us/library/bb924375.aspx
You could emulate typical user behaviour and use one of the cloud services to simulate a huge number of users on the website to see how well your website handles with huge numbers of users. I heard Amazon's service is decent
I can recommend two books published in 2010:
The first is "ASP.NET SITE PERFORMANCE SECRETS" by Matt Perdeck, was published in late fall 2010. It is written more from performance optimization standpoint, but also has detail material on load testing. It is a free pdf eBook.
The second book is ".NET Performance Testing and Optimization - The Complete Guide", by Paul Glavich, Chris Farrell". It is pretty complete source on performance / load testing
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I am in the process of measuring the bandwidth requirements and how a Web application behaves in terms of response time and memory requirements when the number of users is increased.
Is there a particular good tool that can help us here? I believe JMeter is the standard tool. But are there other tools considering that the site is IE only.
Your answer is extremely appreciated.
Well, the question is how do you want to profile? Do you want to simulate real-world activity? Or do you just want to bombard the heck out of the site?
Load Testing Individual Pages
I don't think you can go wrong with using Apache Bench (ab). It's dirt simple to use, and can really stress your application. My typical usage is:
ab -c 10 -n 1000 http://www.example.com/path/to/page
The -c parameter is the number of simultaneous requests to issue. I would suggest starting low (like 5 to 10) and working your way up. Watch the output for failed requests and falling response rate. You're limited to about 1000 connections on most linux machines, so don't go too crazy.
The -n parameter is how many requests to issue. I would suggest doing at least 100 times the number of concurent requests to get a good average...
Another great use for apache bench is to benchmark individual database queries. Just create a simple script that runs the query, and load away. This can be a really good way to detect fast but expensive queries that will take your server down in production yet seem fine in testing.
Load Testing The Whole Application
I've had good luck with WebLoad. There's an Open Source version if you don't have a good budget that will get you started. But I'd suggest springing for the pro version. With it, you can setup a distributed test environment (as simple as installing the client on every machine in the office, as complex as spinning up a bunch of VMs for it).
The cool thing, is that you can program it in javascript. So you can tell it to take random click paths through the site with random delays. This should simulate a user far better than you could do manually. Then, once you have it setup, push the tests to the distributed engine and hit go.
It supports many different load profiles (stair-step where it adds load little by little for the duration of the test, etc). So you can simulate a slashdot-effect profile, normal day-to-day usage, etc.
The reports it generates are immensely useful. It shows you the slow urls, where the bottlenecks are, etc.
There are plenty of other test platforms and systems out there. This was just one that I found that I felt worked pretty well at the time (I did a comparison about 2 to 3 years ago). I am not affiliated with the company in any way.
Load Testing parts of the application
This is a really useful technique called profiling. The how to and tools are fairly language specific, so I won't go into too much detail here (since you don't have a language tag on your question). But the point is that once you find a slow page, you'll need to profile it to figure out what's slowing it down. Then fix the low hanging fruit (the parts that are the slowest). Then re-test to see if you made a difference or not...
Conclusion
Since it's almost impossible to simulate real-world load, this is really more of an art than a science. Have at it, and have fun. Don't take the results to seriously though, even with the best testing, you're likely to miss something... So I wouldn't take them as gospel and go telling the CEO that you tested that it's capable of 100k concurent users. Since when the day it crashes happens (and if you are lucky it will crash), he will blame you since you told him it would work...
Just a thought, you say IE only so is it hosted on IIS? if so then you might want to look at Microsft's WCAT (Web Capacity Analysis Tool), more information is available here:
http://support.microsoft.com/kb/231282
Although it isn't open source but it is free - do you need the source.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
We are "upgrading" the systems at the company, moving from SourceSafe/BugNet/... (yeahy!) to some more serious systems. TFS is too expensive. We have come down to comparing OnTime vs Gemini. They both seem OK with an "OK" price-tag. We will of-course download and try them out both, but it would be nice with comments from experienced users. To me, they seem quite equal.
Has anyone used both, and can compare the two against each-other?
If you would recommend one of these, which one, and why?
Any other experiences with these systems? (Especially Gemini, seems hard to find reviews regarding this-one..?)
(We are talking about a smaller dev-team, max 8 dev in a project at a time, a couple of testers and some stakeholders/managers etc... Several projects running simultaneously. Need to be able to integrate to Visual Studio, Subversion with feed-back to the issue tracker etc)
Thanks for your time!
We use OnTime here - it has a good workflow environment but be warned that it does not scale well at all. We currently have 35 licenses with only 10 to 15 users online at any one time and it struggles.
Also be careful of the sales pitch for using the web or "remote" servers for distributed environments - it works fine with the demo/eval database but slows to a crawl once you start getting a decent amount of items in the database. All you have to do is look at a SQL profiler and you'll see the number of calls made to the DB.
If you profile the web services you'll also see that the web and remote environments have not been optimized at all to batch calls, so as soon as you move to an environment where there is any kind of communication latency it crawls.
Axosoft's support has been less than helpful on this as well - they strangely do not view these as bugs and instead view this as something we should expect in these environments. We have contacted their support over a number of other things as well and it is surprising how poor it is over other things as well.
Axosoft's excuse is that we should have found these things out in the eval period, but I don't know how they expected us to scale the data to production environment levels within a 30 day eval period...
We have been forced to revert to using the WinForm client over Citrix for our distributed teams.
Overall - it is a nice application if you have a small team in a single location. But if you have larger teams or people spread out in multiple locations I would avoid it at all costs.
Gemini is great, we selected gemini over many other bug tracking systems...
main features we liked:
user interface
extensibility (API's, REST based)
addon products (visual studio, outlook plugins)
source control integration (subversion)
source code available (asp.net c#), easy to setup and great support.
For a broader comparison with Gemini bug tracker and others in the same space, wikipedias Bug tracking comparison page might be of use. Although I don't know of a direct, in depth Gemini / OnTime comparison.
Take a look at Project Kaiser (short demo available). It is fast, web-based, supports embedded wiki, forums and chats. And it is free for 5 users :)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
Does anyone have recommendations for a good task/time management tool.
Ideally this would just keep track of programming tasks / the project and the time spent also the option to produce an end of week report would be advantageous
at the lower end, for free software, there's ToDoList
FogBugz is your friend too. FB6 is a great PM tacking tool, I've been using 5 to manage my team and I'm trying to pry the money for FB 6 out of those that hold the purse strings.
For open source/ 1 dev its free as well.
A good tool with collaboration feature as well:
http://www.producteev.com/
We have been using Eylean for our project management. It has all the features you are looking for and then some. You will be able to visualize your tasks as post it notes on a task board, estimate how much time they will take and then track them in order to see how much time they actually take up. It might be little more than what you are looking for, but i really enjoy the software.
JIRA has a log work feature. You log your work on a per task/issue bases. My boss rather likes the reports.
If you work with a large team, Team Foundation has it all. Work item tracking (which can be tied to checkins and builds!), reports, all sorts of metrics. For a small organization though the cost is probably too high.
Voo2do is pretty good. I've used it before and it's great because it's very simple to start, but you can use the extended functionality as needed. And there's an API, which can fill in the missing features.
organize tasks by project
track time spent and remaining
add tasks by email
publish task lists
as easy as paper, but on the web 24x7
supports software guru Joel Spolsky's Painless Software Scheduling method
fancy-shmancy “ajax” interface
API for custom applications
For basic commandline task tracking I use http://wtime.sourceforge.net/ It's simple but effective but only has top level tasks.
Here's another vote for FogBugz. Originally I bought it for managing client projects, but I've grown to use it for keeping track of almost everything to-do related in my life -- simple tasks, stuff to remember, small personal/home projects, books to read, etc. It's feature-rich and has a handful of nice integration points, and while it's still very much software-type-task-oriented, it's handy. Very reasonably priced for the value.
For task management for myself - Outlook
It's centralised
I have to have my email open so I don't need to worry about yet another application
Sync to server so I can get it anywhere I have a browser
Sync to phone so I get reminders when not at the desk
Emails can be flagged as tasks
For task management for a team - TFS 2008 with SP 1. There is some nice enhancements in there with alerting that makes a big difference.
We use Smartsheet as a tasking and collaboration tool. It's pretty well based on the idea of a spreadsheet that you can add your own columns to. The sheets can be shared amongst team members, it sends you emails when anything changes, it has file share (without limit) - you can also give external people access so that they can raise bug notifications etc. that then get notified to the team. Works very well for us as it is SO flexible
I recommend Task Coach, an open source, single person, hierarchical task manager that allows for time tracking, categories, reminders and much more.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
What advice would you have for someone who wants to start in the BI (Business Intelligence) domain?
I where and what I should start with: Books, Blogs, WebCasts...
What I should pay attention to and what I should stay away from.
Are the Microsoft technologies worth while ?
The MS technology stack is quite good and is by far the most accessible (try to get hold of a copy of Cognos Reportnet for self-learning). Where you will run into trouble (and this is the main barrier to entry for gaining a B.I. skillset) is to actually get experience working with real data. It's quite hard to come up with a realistic toy scenario for this sort of thing.
This means that you have to overcome the chicken-and-egg problem that this poses. One option would be to try to get a job as a B.I. developer somewhere like a government department or other place that has trouble recruiting due to salary constraints. Clear evidence of technical skills and a demonstrated interest in the business might get your foot in the door.
This will be a bit harder in a recession. However there is still an ongoing skill shortage of good B.I. people. The reason is (IMO) not the lack of technical skills (the technology isn't rocket science). Instead, I think it is the aforementioned chicken-and-egg problem and the fact that the B.I. domain involves customer intimacy to do it well. It lends itself to working in an analyst/programmer mode with direct customer contact (one of the reasons I do this type of work). If you like working in this mode it might be a good line for you to get into.
Edit: Someone who's just had a job offer in this space asked whether he should take the job.
I found the "project real" from microsoft really helpful while getting into the bi-world. Its a real world bi project, supported by microsoft, to develop and show best practices regarding to all the areas of bi like etl, data warehouse design, cube design, etc.
Business Objects http://www.businessobjects.com/ are quite a big player in this area and familiarity with their products will also help you break into B.I. roles.
For practise data, I would recommend something like the anonomised search records from aol that came out a couple of years back - http://www.techcrunch.com/2006/08/06/aol-proudly-releases-massive-amounts-of-user-search-data/ This is real world size and is an interesting database with some published search sets.
I would stress you to read this book; might seem kind of outdated but the same theory still applies today. It is probably the best starter for general BI.
The Data Warehouse Toolkit - Ralph Kimball
Regarding Microsoft's BI it is a medium-sized tool that can do the job in your first steps (I have more experience with Cognos though). Haven't used MS tools since 2005 so I can't tell much about it.
In case you happen to be interested in Cognos, I have a few videos which can be of help: Cognos Tutorials
Good luck with your project.
Get the Kimball Books (specially this one http://www.amazon.com/The-Data-Warehouse-Toolkit-Dimensional/dp/0471200247) and for starters you may want to start with the MS BI Framework The Microsoft Data Warehouse Toolkit and the SQL Server Enterprise (MS BI Bundle with the database, ETL and reporting), it's easy a readily available, specially if you are a student with the MSDNAA, you can get the enterprise version for free!!!
For general business intelligence, I found the Kimball Group as a great source for best practices.
If you would like to start building your own project, check out GoodData Platform. We have full BI stack Platform as a Service and you can start for free (evaluation period) with access to all resources and learn from Tutorials on our Developer Portal.
I would say try to find a few classes. Microsoft technologies are worth the time. There are many large companies running on the .Net framework.
We use this to get a feel of Microstrategy: http://www.teradatauniversitynetwork.com/apply-and-do
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I'm looking for a good template on server-side installation of software for a project I'm working on.
The client-side is pretty straight-forward. The server-side installation is a little trickier. It is made up of several pieces (services, database connections, dependencies, ports that need to be unblocked, etc.). During a recent test, several undocumented pieces were discovered. Now I need to create installation documentation for our disaster-recovery plans and ways to test the installation without necessarily having a "full-up" system to test on.
I'd really like a suggestion of where I can get a template or a really good example of such a document. I'd like it to be something that an operator could read and comprehend in the heat of a recovery.
[EDIT]
Our current documentation comes mainly from the questions our administrators have had during off-site tests. As new code is written, I'd like to make sure the documentation is written ahead of time. I've been collecting VMWare images to start testing, but was looking for some good examples. It's a Windows Server shop (2000 & 2003). Word templates would be great, but if I could see good documentation, I could create the templates. Any suggestions about what should be tested would be great as well.
[2nd EDIT]
I've gotten several good ideas from the answers posted. After changing my Google search, I came up with some good starting points. They're not perfect, but they are a good start.
Microsoft Exchange - http://technet.microsoft.com/en-us/library/bb125074(EXCHG.65).aspx
iPhone - http://manuals.info.apple.com/en_US/Enterprise_Deployment_Guide.pdf
http://www.novell.com/documentation/gwgateways/gw7_exch/index.html?page=/documentation/gwgateways/gw7_exch/data/ab32nt1.html
http://cregan.wordpress.com/2006/06/22/exchange-2003-step-by-step-installation-instructions/
http://technet.microsoft.com/en-us/magazine/cc160942.aspx
Covers planning in the design stage well - http://www.onlamp.com/pub/a/onlamp/2004/04/08/disaster_recovery.html?page=2
[Edit 10/29/2008]
THIS is the type sample I was looking for. It doesn't have a lot of garbage, but seems to explain enough of the why along with the how http://wiki.alfresco.com/wiki/Installing_Labs_3_Nile
The most complete method that we've come up with for creating our DR documentation, involves going through a full cycle (or two) of installation, and documenting each step along the way.
I realize this can be a bit difficult if you don't have a test (or replacement) system to use to create your documentation - but it's worth lobbying for running through this cycle at least once.
(I recommend twice, the second being done by someone not involved with the project - this is how you test the documentation for future admins, who may not be as experienced with the process.)
A side effect of the above is that your documentation grows fairly large - last I had to do it, I believe the completed installation manual for our database servers was 30+ pages.
What should be tested? Well, in the case of a web site, "can you get to the page?" Include a URL as a starting point and let the admin click through to a certain point. It is not necessary for the admin to go through the whole QA cycle, just a confirmation that what you meant to be deployed is really what got deployed.
Other ideas
Also, we (my team at my last job) had QA test the deployment. As a QA person should be, he was not intimate with the details and as he deployed to QA, we were able to get feedback on what went wrong.
Another thing that is useful is sitting down with the admin(s) before the deployment. Go over the instructions and make sure they understand them the same way you do.
Template? Just make sections that have fields for data such as URL to DEV, QA, and PROD. When you write out the instruction you can refer to those. Just make it clear what is being deployed.
Depending on the admins, automation is helpful. I've had windows admins that want a Word doc with step by step instructions and other admins that wanted a script.
However, some helpful things to include, probably as sections
Database changes
Scripts to run
Verification that they worked
Configuration changes
what are the change
where is a version of the new file (In my case they diffed the two, which helped reduced errors concerning production-specific values)
General verification
what should be different from the user perspective (feature changes)
For web farm deployments, it might be helpful to have a coordination document concerning how the servers need to be pulled in and out of pool.