I have a Drupal site with a lot of calculations and database requests on each page load (running on an Amazon EC2 server). I am curious how my site would hold up if it became popular or in some other way received heavy traffic. Perhaps the most important thing for me is to locate potential bottlenecks in my code.
What are the best tools for stress testing and finding bottlenecks in a Drupal site? Right now I'm not using any cache module. I've read about the MemCached module and some about Varnish.
Anyone who can share their experiences?
Generally php does not do great in really large scale projects. That is why google does not support php in their app engine.
Facebook which was made in php had to be compiled into c++ for better performance.
Having said that here are the some of the tools ( I have not used them )
http://www.webload.org/
http://xdebug.org/ - to profile your php code besides debugging
cheers,
Vishal
Apache Benchmark.
Send no cookies to simulate anonymous traffic and stress the Varnish caching layer.
Send rando cookie to stress the Drupal caching layer (where you are hopefully using memcache)
Send a login cookie to stress the DB layer.
We run massive Drupal sites on php. Scaling does cost in resources for multiple webheads, database clustering, separate memcache and file servers, but you have to balance that cost against hiring developers for refactoring your code into a different language, and code maintenance.
Related
There is a lot written about the performance of Laravel. It is not the very best framework when it comes to fast applications but it has a lot of options and a brilliant community and documentation. I would like to know if Laravel suits my situation:
I'm currently developing a browsergame that, hopefully, will be played by thousands of visitors worldwide. There could be over 3000 concurrent users at the same time. The application is a bit heavy, because it needs a lot of different modules: views, routing, session management, authentication, database connections, cronjobs and so on. It really is a dynamic game so the application will be loaded a lot of times. And: I don't have that much money to invest in a lot of dedicated servers (at least not in the very beginning).
I looked to other frameworks too. Because Lumen, Slim and some other micro frameworks don't support all of the modules my game needs, I think Laravel is a good choice. But I'm really scared for the benchmarks I see. Laravel doesn't look good there when compared to other frameworks: it's slow, consumes a lot of memory and can't handle many requests.
So my question: is Laravel a good choice for a heavy browsergame website with potential of thousands of concurrent users? Caching and Homestead will help for sure, but is Laravel a good choice at all, or is there a really better framework?
It's less about the framework, and more about how you write your code.
Follow good practices, plan for scaling, and Laravel can absolutely be a very high-performance solution for you. I've heard of some using Laravel with millions of requests per day.
We run a large production app with Laravel, load balanced across multiple web servers, separate redundant database servers, Redis caching, etc. We've had lots of scaling issues, but interestingly none of them were related to the framework. Your main bottlenecks will be elsewhere.
Folks worry way too much about performance before even beginning a project (premature optimization, and all that). Pick the best tool for the job in terms of what it does for you. Then build your app with scaling in mind.
If Laravel provides the functionality you need and you like how it works, use it!
I've just added a CDN distribution using Amazon Cloudfront to my Rails application on Heroku, it's working OK.
My homepage serves around 11 static assets, I've made some tests using http://www.webpagetest.org/ and there are no differences (in terms of performance, optimizing load times) between using the CDN or not.
Is there any particular reason why this could be happening?
My region is Latin America btw, so it's using the All locations edge option.
Thanks.
The main benefits of using CDN from Amazon or others is that they are hosted on fast and reliable servers and offload the traffic served directly from your server, which in case that you have a dedicated fast server you won't see a considerable boost.
But another benefit is that they are potentially cached by user's browser (due to visiting other websites which have used the same CDN) so the visitor will have a better experience first time they visit your site.
A couple of suggestinos.
If the site CSS is one of the static assets that you have moved to CloudFront then I would try moving it back to your main server.
Since page display can't start until the site CSS is downloaded, you want to serve this as fast as possible. If it's coming from a CDN then it requires a second HTTP request.
Also, use the waterfall display from webpagetest.org to pinpoint where the bottlenecks are.
Good luck!
Is cloud hosting the way to go? Or is there something better that delivers fast page loads?
The reason I ask is because I run a buddypress site on a bluehost dedicated server, but it seems to run slow at most times of the day. This scares me because at the moment the sites not live and I'm afraid when it gets traffic it'll become worse and my visitors will lose interest. I use Amazon Cloud to handle all my media, JS, and CSS files along with a catching plugin, but it still loads slow at times.
I feel like the problem is Bluehost, because I visit other sites running buddypress and their sites seem to load instantly. Im not web hosting savvy so can someone please point me in the right direction here?
The hosting choice depends on many factors such as technical requirements, growth rates, burst rates, budgets and more.
Bigger Hardware
To scale up hosting operation, your first choice is often just using a more powerful server, VPS, or cloud instance. The point is not so much cloud vs. dedicated but that you simply bring more compute power to the problem. Cloud can make scaling up easier - often with a few clicks.
Division of Labor
The next step often is division of labor. You offload database, static content, caching or other items to specific servers or services. For example, you could offload static content to a CDN. You could a dedicated database.
Once again, cloud vs non-cloud is not the issue. The point is to bring more resources to your hosting problems.
Pick the Right Application Stack
I cannot stress enough picking the right underlying technology for your needs. For example, I've recently helped a client switch from a Apache/PHP stack to a Varnish/Nginx/PHP-FPM stack for a very business Wordpress operation (>100 million page views/mo). This change boosted capacity by nearly 5X with modest hardware changes.
Same App. Different Story
Also just because you are using a specific application, it does not mean the same hosting setup will work for you. I don't know about the specific app you are using but with Drupal, Wordpress, Joomla, Vbulletin and others, the plugins, site design, themes and other items are critical to overall performance.
To complicate matter, user behavior is something to consider as well. Consider a discussion form that has a 95:1 read:post ratio. What if you do something in the design to encourage more posts and that ratio moves to 75:1. That means more database writes, less caching, etc.
In short, details matter, so get a good understanding of your application before you start to scale out hosting.
A hosting service is part of the solution. Another part is proper server configuration.
For instance this guy has optimized his setup to serve 10 million requests in a day off a micro-instance on AWS.
I think you should look at your server config first, then shop for other hosts. If you can't control server configuration, try AWS, Rackspace or other cloud services.
just an FYI: You can sign up for AWS and use a micro instance free for one year. The link I posted - he just optimized on the same server. You might have to upgrade to a small server because Amazon has stated that micro is only to handle spikes and sustained traffic.
Good luck.
It takes about 20 seconds to login to the admin backoffice. During this time load would spike (from 0.02 to 0.20)
Yet I see no "slow queries" in the mysql log. I've optimized all the tables.
Is there some way to hunt down the source of this slow down?
Which browser are you using? The Magento admin can be very slow in some browsers i.e. Internet explorer. I recommend using Chrome or Firefox for admin work, especially adding/editing products. If you are generally looking at ways to speed up Magento then here is a good place to start:
https://web.archive.org/web/20140327173649/http://www.gxjansen.com/101-ways-to-speed-up-your-magento-e-commerce-website/
The admin panel pulls in a lot of files and is heavily javascripted. Turning on things like mod_deflate and mod_expires can help tell your browser to not download these files as often.
Magento is extremely resource intensive and the admin interface does not use any caching. A lot of it depends on the hardware your site is on. For any site is decent traffic and heavy admin use, the base line should be a dedicated dual quad core server with 24-32mb of ram, especially if you have the db and web on the same server. Any shared server will just not cut it. Cloud based solutions vary, we have had some good results with splitting db and load balanced web in a cloud environment. Make sure you have apache and mysql properly tuned based on the memory of the server and use a php accelerator like apc.
What various methods and technologies have you used to successfully address scalability and performance concerns of a website? I am an ASP.NET web developer exploring .NET remoting with WCF with SQL clustering and am curious as to what other approaches exist (such as the ‘cloud’). In which cases would you apply various approaches (for example method a for roughly x many ‘active’ users).
An example of what I mean, a myspace case study: http://highscalability.com/myspace-architecture
This is a very broad question making it difficult to answer, but I'll try and provide a few general suggestions.
1 - Unless you are doing some things seriously wrong then you'll likely not need to worry about perf or scale until you hit a significant amount of traffic (over 1 million page views a month).
2 - Your biggest performance problems initially are likely to be the page load times from other countries. Try the Gomez Instance Site Test to see the page load times from around the world, and use YSlow as a guide for optimizing.
3 - When you do start hitting performance problems it will first most likely be due to the database work. Use the SQL Server Profiler to examine your SQL traffic looking for long running queries to try optimizing, and also use dm_db_missing_index_details to look for indexes you should add.
4 - If your web servers start becoming the performance bottleneck, use a profiler to (such as the ANTS Profiler) to look for ways to optimize your web pages code.
5 - If your web servers are well optimized and still running too hot, look for more caching opportunities, but you're probably going to need to simply add more web servers.
6 - If your database is well optimized and still running too hot, then look at adding a distributed caching system. This probably won't happen until you're over 10 million page views a month.
7 - If your database is starting to get overwhelmed even with distributed caching, then look at a sharding architecture. This probably won't happen until you're over 100 million page views a month.
I've worked on a few sites that get millions/hits/month. Here are some basics:
Cache, cache, cache. Caching is one of the simplest and most effective ways to reduce load on your webserver and database. Cache page content, queries, expensive computation, anything that is I/O bound. Memcache is dead simple and effective.
Use multiple servers once you are maxed out. You can have multiple web servers and multiple database servers (with replication).
Reduce overall # of request to your webservers. This entails caching JS, CSS and images using expires headers. You can also move your static content to a CDN, which will speed up your user's experience.
Measure & benchmark. Run Nagios on your production machines and load test on your dev/qa server. You need to know when your server will catch on fire so you can prevent it.
I'd recommend reading Building Scalable Websites, it was written by one of the Flickr engineers and is a great reference.
Check out my blog post about scalability too, it has a lot of links to presentations about scaling with multiple languages and platforms:
http://www.ryandoherty.net/2008/07/13/unicorns-and-scalability/
There is velocity from MS as well as MEMCache has a port to .NET now and also indeXus.Net