I manage a database (Oracle 8i) and web server (IIS) for about 50 simultaneous users on average and a theoretical limit of 100 simultaneous users. A mid level system.
We just got a dual-socket Quad-core XEON - 16GB RAM - SAS-RAID-10 beast and I am exploring the possibilities of taking these two separate servers and merging them into two virtual machines both running on the new server (Server 2009 Hyper-V).
1) In general, what are the performance penalties (as well as any gotchas and hidden consequences) of running both the database and web servers as virtual machines on one mega server vs running them on two separate slower boxes? Is it a big NO-NO or it is something worth trying for a mid-level system that will never need to scale?
2) What are the general performance penalties (in percentage) and gotchas for virtualizing just the database server? We run Oracle 8i (but are considering moving to MS SQL Server).
3) If only stress tests can determine an reasonable answer, what would be the easiest way to test these scenarios (tools / configuration).
Thanks in advance for any generous knowledge-sharing.
If you are looking to do this, I would check Microsoft's site and best practices on how to do it. There is a podcast on Deep Fried Bytes that talks about how the Microsoft.com site is setup to use virtual servers and some of their practices on how they implement it. They don't seem to have performance penalties on how they run it, but I am not certain of the details (it also talks about how they use server virtualization like a real organization would and not a company with unlimited amounts of money to throw at a problem).
I believe this is the podcast:
http://deepfriedbytes.com/podcast/episode-8-behind-the-scenes-at-microsoft-com/
With regards to databases, see this question:
Virtualized SQL Server: Why not?
Note that this is specific to sql server, but many of the same principles will apply for oracle.
As for web servers, virtualization is a great idea. It can make it easier to increase reliability and scalability.
I think at your level of concurrent user connections, and the power of the machine, you won't have too many performance issues running SQL Server on a VM.
We have a mix of VMWare ESX VMs and bare-metal OS' running app, web, and DB servers and without a doubt the heaviest loaded DBMS system is on bare-metal machine (Quad proc quad core, etc.). All the little guys, though, live on VMs, and we haven't noticed any problems (even using iSCSI over GigE).
One thing to consider is you won't get any fault tolerance out of a single setup like this because a CPU failure will bring down the entire box, thus blowing up your whole app.
More info on SQL Server HA and Hyper-V, just FYI:
http://blogs.technet.com/andrew/archive/2008/11/10/sql-server-2008-hyper-v-and-high-availability.aspx
Be aware that Oracle has its own guidelines on running in a virtual machine.
The product I work with utilizes Oracle on the back-end, and for heavy use, the overhead of a VM has had negative effects on it.
8i is well past EOL, and was around before virtualization was a Big Thing(tm), so moving to a new edition of Oracle might also be a good plan at the time your migrate to virtualization.
Oracle blog article on 11g in a VM - http://blogs.oracle.com/MingMan/2007/11/oracle_database_11g_successful.html
If you're concerned about timing, also be aware of known clock-drift issues in hypervisors, and available fixes (either from the OS or virtualization vendors).
I recently came across an article dealing with Virtualization Security. I thought it would be worth mentioning here.
Related
I am evaluating a system architecture where the application is split on two virtual servers on the same hardware.
The reason is that the overall system will perform better if a certain part of it runs on its own virtual server. Is this correct?
I would think that if the processes run on the same hardware, a split on two servers will give an overhead to communication compared with installation on one virtual server.
To make sure, it sounds like you're asking, more or less, about how 2 (or more) virtual machines might "compete" with one another, and how might separating the processes run on them affect the overall performance?
Quick Answer: In short, the good news is that you can control how the VM's "fight" over resources very tightly if you wish. This will keep VM's from competing with each other over things like RAM, CPU, etc. and can improve the overall performance of your application. I think you're interested in knowing about two main things: VM reservations/limits/shares and resource pools. Links are included below.
In-depth Answer: In general, it's a great idea to separate the functionality of your application. A web server and a DB server running on different machines is a perfect example of this. I don't know about your application in particular, but if it's not leveraging multi-threading (to enable the use of multiple processors) already, separating your application onto two servers might really help performance. Here's why:
VM's understand, in a sense, what HW they're running on. This means they know the CPU/RAM/disk space that's available to them. So let's say your ESX server has 4 CPU's and 16 GB of RAM. When you create a VM, you're free to give 2 CPU's and 8 GB of RAM to each server, or you can alter the setting to meet your needs. In VMware, you can guarantee a certain resource level by using something called: limits, shares, and reservations. Documentation on them can be found here http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.vmadmin.doc_41/vsp_vm_guide/configuring_virtual_machines/t_allocate_cpu_resources.html, among other places. These reservations will help you guarantee that a certain VM always has access to a certain level of resources and will keep VM's from causing contention over RAM, CPU, etc. VMware also offers another extension of this idea using something called "resource pools", which are pools of RAM, CPU, etc. that can be set aside for certain machines. You can read about them here: http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.resourcemanagement.doc_41/managing_resource_pools/c_why_use_resource_pools.html.
Is there any reason in particular why it's recommended to run memcached on a Linux server? Is it really that bad an idea to run it on a Windows Server box? What about an OS X Server box?
The biggest reason that I read is about TCO. In other words, for each windows box that we run memcached on, we have to buy a copy of Windows Server and those costs add up. The thing is that we have several servers that have older processors but a lot of RAM - perfect for memcached use. All of these boxes already have Windows Server 2003 installed on them, so there's not really much savings to installing Linux. Are there any other compelling reasons to use Linux?
This question is really "what are the advantages of Linux as a server platform" I'll give a few of the standard answers:
Easier to administer remotely (no need for RDP, etc.) Everything can be scripted or done via CLI.
Distributions like the Ubuntu LTS (Long Term Support) versions guarantee security updates for years with zero software cost. Updates can easily be installed via command line, and generally don't require a reboot.
Higher performance. Linux is generally considered to offer "more bang for the buck" on a given piece of hardware. This is generally due to lower resource requirements.
Lower resource requirements. Linux runs just great on 256MB or less of RAM, and on very small CPUs
Breadth of available software & utilities.
It's free. (As in beer)
It's free. (As in freedom) This means you can see, change, and file bugs against the code you're running, and talk directly with the developers.
Remember, that TCO includes the amount of time that you (the administrator) are spending to maintain the machine. Linux has a lower TCO because it's easier to maintain, and you can spend your time doing something other than administering a server...
Almost all of the FAQs and HOWTOs are written from Linux point of view. Memcache was originally created only for Linux, the ports came later. There is port to Windows, but it's not yet in the official memcache distribution. Memcache on Windows is still guerrilla style. For example there is no memcache for x64 Windows.
As of memcache on MacOS X on servers: niche of niche of niche.
There doesn't seem to be any technical disadvantage to running it in windows. It's mainly a cost thing. If the licenses are just sitting around unused, there's probably no disadvantage at all. I do recall problems on older windows with memory leaks in older windows APIs, particularly the TCP stuff -- but presumably that stuff is all fixed in modern windows.
If you are deploying memcached you probably have a fairly significant infrastructure (many, many machines already deployed). Even if you are dedicating new machines to memcached, you'll want to run some other software on them for system management, monitoring, hardware support etc. This software may be customised by your team for your infrastructure.
Therefore, your OS platform choice will be guided by what your operations team and hardware vendor will support for use in production.
The cost of a few Windows licences is probably fairly immaterial and you probably have a bulk subscription already - in fact the servers may be ordered with Windows licences already on them.
Having said that, you will definitely want a 64-bit OS if you're running memcached - using a 32-bit OS is not clever and will mean that most of your RAM cannot be used (you'll be limited to around 3G depending on the OS).
I'm assuming that if you're deploying memcached, you'll be doing so on hardware with LOTS of ram - it's pretty pointless otherwise, after all.
I am trying to perform a fair comparison of XenServer vs ESX and one comparison I would like to make is performance with multiple VMs. Does anyone know how to go about benchmarking VM performance in a fair way?
On each server I would like to run a fixed number of XP/Vista VMs (for example 8) and have some measure of how quickly each one runs when under load. Ideally I would like some benchmark of the overall system (CPU/Memory/Disk/Network) rather than just one aspect.
It seems to me this is actually a very tricky thing to do and obtain any meaningful results so would be grateful for any suggestions!
I would also be interested to see any existing reports or comparisons that have been published (preferably independent rather than vendor biased!)
As a general answer, VMware (together with other virtualization vendors in the SPEC Virtualization sub-committee) has put together a hypervisor benchmarking suite called VMmark that is available for download. The VMmark website discusses why this benchmark may be useful for comparing hypervisors, including an FAQ and a whitepaper describing the benchmark.
That said, if you are looking for something very specific (e.g., how will it perform under your workload), you may have to roll your own variants of VMmark, especially if you are not trying to do the sorts of things that VMmark benchmarks (e.g., web servers, database servers, file servers, etc.) Nonetheless, the methodology behind its development should be of interest.
Disclaimer: I work for VMware, though not on VMmark.
I don't see why you can't use common benchmarks inside the VMs: WinSAT, Passmark, Futuremark, SiSoftware, etc... Host the VMs over different hosts and see how it goes.
As an aside, benchmarks that don't closely match your intended usage may actually hinder your evaluation. Depending on the importance of getting this right, you may have to build-your-own to make it relevant.
Why do you want to bench?
How about some anecdotal evidence?
I'm going to assume this is a test environment, because you're wanting to benchmark on XP/Vista. Please correct me if I'm wrong.
My current test environment is about 20 VMs with varying OS's (2000/XP/Vista/Vista64/Server 2008/Server 2003) in different configurations on a Dual Quad Core Xeon machine with 8Gb RAM (looking to upgrade to 16Gb soon) and the slowest machines of all are Vista primarily due to heavy disk access (this is even with Windows Defender disabled)
Recommendations
- Hardware RAID. Too painful to run Vista VMs otherwise.
- More RAM.
If you're benchmarking and looking to run Vista VMs, I would suggest putting your focus on benchmarking disk access. If there are going to be performance differences elsewhere I doubt they would be of anything significant.
I recently came across VMware's ESX Performance documentation - http://www.vmware.com/pdf/VI3.5_Performance.pdf. There is quite a bit in there on improving performance and benchmarking.
OK we are at the end of our rope here, and I’d really appreciate feedback from the SO community.
Our basic issue is slow performance by our MOSS-based intranet--
Some environment info:
We have a MOSS standard edition for a collaboration based site.
The sitedb is 29 Gb
we have two VMWare based front end
servers. (2x 32bit CPUS each)
less than 1000 users spread over all
timezones
We have one big site
collection with subsites.
General symptoms:
Loading front page and pages that have been ‘warmed-up’ is pretty decent- but pages/sites off the beaten path are very slow to load.
We see spikes, where a page all of a sudden takes 30 seconds to load, vs its more normal 2
Here’s what we have done already:
Scaled way back on crawling
enabled object and blob caching
optimized VMware setup
followed the Microsoft IT whitepaper on MOSS sharepoint best practice (esp list size etc)
I don’t know what else to do here—Split into multiple site collections?
Switch to 64 bit front-end servers?
Would be great to hear from others who have been in similar situations.
You don't say how much memory your front end servers have - given that they are 32bit, I'll assume the maximum per worker process of roughly 2gb + change.
My advice? Switch to 64 bit, add more memory, and check that you are not using just one w3wp worker process per front end. Have a dig into "web gardens," that is to say where you configure multiple w3wp processes per front end. To start with, start with two workerprocesses per front end and see how that works out. Also make sure they are set to recycle, and that the recycling of each pair of worker processes do NOT overlap - having two+ workers means they can take turns to recycle without cutting access.
just my 0.02.
-Oisin
I think your very first task is to determine where the problem actually is -
until you know that you are wasting your time changing things.
Is the database server on a separate server or one of your web servers?
Do you see a CPU/Disk bottleneck on your front end or db servers?
It sounds like your world wide; do you see the same performance problems from networks close to the server - is it a WAN issue?
Thanks for some helpful advice all, one thing I just learned was that our object caching has basically not been doing anything! This is because the way it seems to work is that if you have rights to edit ANYWHERE on the site collection, it per default disables object caching across the portal. Since all users have rights to at least something, this means caching was doing basically nothing!
We discovered this by enabling the cache debugging, which puts a small comment in the html about what cache is being used. After changing the setting "Allow writers to view cached content" in the authenticated cache profile,
We are seeing what this does for editors, but for regular viewers, the anecdotal evidence is that it is having a big impact!
Yes, caching is the best way to reduce load on the system.
Adding RAM to the SQL server is also good. (64 bit is really a must for your SQL server, WFE not so important).
Not sure if you want to recycle the processes though. I have not evidence for this except a conversation with someone saying the recycling the processes looked to have solved one performance issue, but was introducing others.
Did I mention caching?
The SQL server should handle database up to 100Gb, but at that size they will be hard to manage for backups and the like, so splitting your site into relevant site collections is something you may need to plan for now, but this may not be relevant to performance.
Did you take a look at Plan for software boundaries (Office SharePoint Server)?
At first glance, your server fits in their recommended settings.
To improve performance you should take a look at :
64-bit servers
Limiting the number of items displayed in your document lists
(source: microsoft.com)
Definitely check the disk usage. If you have two VMs and they run of the same disk / SAN, make sure it isn't too busy. Overloaded SANs kill performance
I'll throw my hat in the ring and recommend 64 bit as well. Perhaps not as an immediate fix, but going forward, I would put a move to 64 bit as a goal for your entire farm.
I'd also take some issue w/ Nat's comment that it doesn't matter on the Web Front Ends. I won't debate benchmarks or argue memory addressability. It's really simpler than that.
Microsoft has publically stated that 2007 is the last version of SharePoint that will run on 32 bit servers. Going forward 64 bit is going to be a requirement - so like the FRAM oil filter guy says - "pay me now or pay me later"...
we have also performance problems. We upgraded on win 2008 64bit witch make some difference but not as much as expected.
The bih boost gave us changing from NTLM azthentication to Kerberos. This was our major improvement.
hope this helps to someone
I run a very similar setup, however we have over 400GB spread over 3 site collections. There are a few other things you can attempt before going down the 64bit path.
Ensure there is no Disk or Bandwidth issues between the DB server and web front ends. The network speed to the database server is critical.
Check the database server itself, if the disks are on a SAN there may be performance issues if there is contention for the phsyical drive on the san. RAM is also important, the more it can cache to memory the less time waiting for disks to respond.
Schedule crawling jobs to run outside of regular business hours
You've enabled object caching which can be huge help, be sure to ensure compression is enabled too! For lower bandwidth users sharepoint sends a large amount of CSS information to users that compresses down to 1/10th it's size if compression is enabled.
Heres another big one, ensure your companies DNS is working correctly in other locations and look into if this problem is related to external sources. IE we have a sonicwall firewall that is brutal on response times for offices connecting through it for anything.
Micrsoft has a whitepaper on performance counter monitering. It is very thorough and will help you narrow down the problem between CPU/RAM/Network/HD IO.
Hope this helps!
I don't expect a straightforward silver bullet answer to this, but what are the best practices for ensuring good performance for SharePoint 2007 sites?
We've a few sites for our intranet, and it generally is thought to run slow. There's plenty of memory and processor power in the servers, but the pages just don't 'snap' like you'd expect from a web site running on powerful servers.
We've done what we can to tweak setup, but is there anything we could be missing?
There is a known issue with initial requests once an IIS application pool has unloaded the SharePoint resources or recycled itself where the spin-up on a new request is very slow.
Details about why that happens and how to fix it can be found here; SharePoint 2007 Quirks - Solving painfully slow spin-up times
Andrew Connell's latest book (Professional SharePoint 2007 Web Content Management Development) has an entire chapter dedicated to imporving performance of SharePoint sites.
Key topics it covers are Caching, Limiting page load (particularly how to remove CORE.js if it's not needed), working with Disposable objects and how to work with SharePoint querying.
2 really good tricks I've got are to use the CSS Freindly Control Adapters to generate smaller HTML for the common components (menus, etc) and setting up a server "wake up", so when IIS sleeps the app-pool due to inactivity you can reawaken it before someone hits your site.
Microsoft has released a white paper on this very issue.
How Microsoft IT Increases
Availability and Decreases Rendering
Time of SharePoint Sites Technical
White Paper Published: September 2008
Download it from here.
SharePoint has a lot of limitations that contribute to low performance problems we may call them performance bottlenecks. The SharePoint performance problems occur primarily due to the following reasons:
BLOBs overwhelm SQL Server
Too many database trips for lists
You can dramatically improve SharePoint performance if you use a few of intelligent techniques which are:
Externalize Documents (BLOBs)
Cache Lists and BLOBs
Microsoft Office SharePoint Server (MOSS) is an extremely popular product that improves effectiveness of organizations through content management and enterprise search, shared business processes, and information-sharing across boundaries for better business insight. And StorageEdge is an extremely fine product that enhance/improve SharePoint performance.
By using StorageEdge SharePoint's performance can easily be enhanced.
Just a few ideas...
Is displaying your pages as slow when doing it from the Server or from a Client? If slower from client, do check your network.
Are your pages very "heavy"? (means, many elements, web parts and so on?) Than maybe it's normal.
Have you noticed they load slowlier since you've add one specific web part? Maybe there's some issues with that specific web part (for example, it's accessing to a document library that has many -thousands of- documents). If that's the case, try to deactivate that specific web part and see if the performance works better.
I noticed that Sharepoint loves to add a ton of JavaScript. If you run a Browser with slow JavaScript (say, Internet Explorer), i notice that it sometimes does not "feel" fast.
Also, if you are running custom code on it: Make sure to dispose your SPWebs after use, that can up a lot!
Are you running on virtual or physical servers? We found that it is significantly faster on physical servers. Also, check the disk performance - if you are running the servers from a SAN it might be a sign that your SAN is over utilised.
To investigate SharePoint performance issues, I would try these things first, in that order:
Run SQL profiler for those non-performing pages. SharePoint API excels at hiding what's going on behind the scenes in respect to database roundtrips. There are single API calls that, without the knowledge of the developer, generate many roundtrips, which hurt performance.
Profile w3wp.exe process serving your SharePoint site. That's going to tell you relative API usage. Focus on ticks, no time, and do a top-down inclusive time analysis to see which calls are taking up most of the time. See here for instructions.
Run Fiddler or Microsoft NetMon to spot potential excessive client roundtrips (i.e. between browser and web front end server) and redirections (301's).
The three main major components of SharePoint setup is SharePoint Server (the one which runs WSS/SPS services), SQL Server DB and IIS.
You said you have decent power for your SharePoint services and I assume IIS would definitely be on a good machine.
Usually SQL Server setup that hosts the SP related DBs that would slow down the page loads. I would want you to take a look at your whole SQL Server related performance counters and you might want to tune these DBs too (which includes OS Server/Stored Procedures/Network, etc)
Hope this adds to your checklist of things you want to take a look at.