SQL Azure is responding at 90ms avg
IIS hosted in Azure VM is responding at 2000ms avg!
What can it be done to improve the network speed of the Azure VM?
I have a webapi app hosted in an azure virtual machine. This app links to 20 SQL Azure databases using the elastic scale client API.
Response times from SQL Azure are good: 90ms avg on 900 simultaneous users running queries against the webapi. I know SQL Azure is responding quickly because I am logging response times at the controller level in my webapi including json deserialization of the response object.
However, in my load test I'm consistently getting response times of right around 2000ms!
So, there's a 20 time discrepancy between what SQL Azure is doing to what IIS in the virtual machine is returning. The virtual machine network speed is the bottleneck now and I can't figure out how to solve it.
I have looked through several posts and fixed the following:
Ensure power management is set for performance instead of balanced.
However, slow performance is still there.
I have the following setup:
Azure Virtual Machine D3 (4 cores 14GB memory). This is a 500USD per month machine... it should be pretty fast, right??
20 SQL Azure databases at Standard S3 Level. I'm happy with these ones delivering 90ms on 900 users... Actually I think they can support a lot more users than 900 at the 90ms response times avg but
WebAPI from .net 4.5
I'm very happy that through sharding I have been able to improve the performance of my database searches but now the network speed of the virtual machine I'm using is degrading my overall app performance to a point that it makes it a showstopper.
Any pointers on what I might be missing would be extremely appreciated.
Cheers,
Hector
Related
I've noticed on my dev env. timeout sql connection errors when i'm using connection string to remote db.
I've developed a small tool to ping domain and db server based on these answers test if a website is alive from a C# applicaiton
and test SQL Server connection programmatically
When i noticed Failed pings i looked into site management console and caught that Sql Server is unavailable, the site was down for about 5 minutes.
Since i started monitoring the issue repeated 3 times for the last couple of days. It means that my DB server withing a shared hosting plan is not reliable 24/7, i opened a ticket and got a reply from support:
As this is a shared server, the activities on the server always varies from time to time. We apologize if there is a slight issue earlier
Is this a common situation for any shared asp.net hosting? or it is a bad luck and i need to search for another hosting?
Sometimes when the hosting providers update some service or software it could be down for a few minutes, but this should not happen very often. You could continue monitor the services and if the results are not good you could try another hosting provider.
You may experience little slowness or lagging in I/O in shared database servers while database backup script is running in background or any other maintenance are carried out by the web host. But in most cases, they don't affect the server availability.
In fact, shared database servers are really high end servers (mostly SSD base) and are meant to host thousands of databases without single hiccup. They must be capable to handle millions of queries at any point of time. If you face this problem more often then it's straight indication that your web host is overly utilizing the database server resources, or server is no longer capable to handle the load in peak hours.
I want to run an server for an application I have.
I'm a complete beginner with AWS, so bear with me.
There will be about 50 users (all from the same time zone) that will be accessing the server and I would like to have near 100% availability.
The application I have requires 2 processors and 2GB ram.
I could pay for a machine 24/7 or even only 18 hours a day, assuming I turn it off at night, but I there will be some days where the server is not used at all.
I was wondering if the following is possible: when amazon detects that someone is requesting something from my server, it turns it on in real time, and then forwards the request to my server. After say 5 minutes of no activity, it will turn my server off. This way I can only pay for hours when there is traffic.
Is this possible?
How have people solved similar problems?
No this isn't possible. There is nothing built into AWS to detect traffic and start an EC2 server like you suggest. Plus the startup time on an EC2 server is at least a couple minutes, so those first incoming requests would have to wait a really long time.
You might want to look into running multiple small servers instead of a single larger server. AWS does have the ability to balance the load across multiple servers and add/remove servers from the pool based on traffic. You could have as few as one server running when there is no traffic, and have more servers automatically created as load increases. Look into the AWS Elastic Beanstalk service for this.
If you want to run a truly "serverless" environment where you only pay for compute cycles in milliseconds, instead of servers by the hour, you could look into using AWS Lambda. If you can architect your system to run on Lambda you are almost guaranteed to save costs, but it can be a real challenge to convert an existing system to this sort of architecture.
If you want to look outside AWS you might find something more along the lines of what you describe with Google App Engine. Heroku's free tier also works similarly to your description, but as soon as you outgrow the free tier you have to upgrade to always-running instances.
I have a web role with co-located cache. there are two instances of this role.
Even when there is a cache-hit, the turn-around time for our request measures to a few seconds. Upon analysis we found that the time taken by cache to get back with data is 1 second on average. However, IIS logs suggest that the overall servicing of the request takes about 4 seconds. there is no intermediate operation before or after data retrieval from cache.
What could be wrong here? What would be a good way to analyse the problem?
For what it's worth we were having a similar problem with caching in Redis in Azure and a RESTful API.
The problem turned out to be the serialization of data.
Some ways to debug the problem:
Download ANTS profile (it has a free trial) and profile your worker role locally.
Enable profiling for your worker role, deploy it, run it for a bit, then download the profile file in Visual Studio. (You can use Server Explorer to find your instance and download the log).
Download the Azure tool kit (http://blogs.msdn.com/b/kwill/archive/2013/08/26/azuretools-the-diagnostic-utility-used-by-the-windows-azure-developer-support-team.aspx) on your instance. It has things like Process Explorer that can tell you how much memory your role is taking, how much CPU, what it's doing on the network etc.
You can contact Azure support and have them help you profile your application. We did that and got absolutely amazing support. They talked with us on the phone for hours and helped us profile our code.
You really should increase the log level for client and server refer In-Role Cache Troubleshooting and Diagnostics (Windows Azure Cache) and take a look at the performance counters. If read operations (GET) is taking long time then there can be paging in one of the instances or may be there is overload on the server. If you see any performance issue on the cache instances then you should take reassess the capacity using Capacity Planning Considerations for In-Role Cache (Windows Azure Cache) .
If this doesn't help then please open a support ticket.
I have a rather strange scenario. We have a range of WEBAPIs hosted on the cloud. We consume those services in our Windows 8 application. The problem is when the services are running locally it takes less than 400ms but when hosted on Windows azure it takes upto 20 seconds for some requests. I have checked the indexes of our database tables and its fine. I have no clue so as to what to profile and how to improve the performance.
Thanks!
Everyone Thanks a lot!
But I found a way to use dottrace(Excellent profiling tool) on the azure deployment. Here is the link
http://blog.maartenballiauw.be/post/2013/03/13/Remote-profiling-Windows-Azure-Cloud-Services-with-dotTrace.aspx
You can also use windows azure diagnostics and stopwatch class to log all times to the wad tables.
Also found out that the first request to the azure service is always slow in another thread. Have just copied it here below
Serkan, you would need to first make sure in your post, weather you have published a Cloud Service or a Website to Windows Azure. Based on Cloud Service (A Web Role) or a WebSite the answer to your question will be different. As you want to learn more I would explain what goes on behind.
As you suggested that your first connection is slow, I can see that happen with Windows Azure Websites. Windows Azure Websites are running in shared pool of resources and uses the concept of hot (active) and cold (inactive) sites in which if a websites has no active connection for x amount of time, the site goes into cold state means the host IIS process exits. When a new connection is made to that websites it takes a few seconds to get the site ready and working. Depend on how your first page code is, the time to load the site for the first time varies. Similar discussion is logged: Very slow opening MySQL connection using MySQL Connector for .net
With Windows Azure Cloud Service the overall application model is different. Your webrole has its own IIS server which is fully dedicated to your application and above Website limitation does not occur however there could be other reasons which could have slower page load. If you are using WebRole, then what you could do is run a page load profiler first and RD to your Azure Instance to collect the page load data to see what else you could do to boost the performance.
You'll obviously need to profile your app to find the real cause. Check out these two articles which should get you started:
http://msdn.microsoft.com/en-us/library/windowsazure/hh369930.aspx
http://www.windowsazure.com/en-us/develop/net/common-tasks/profiling-in-visual-studio/
The title pretty much says it. Can a micro instance (which only has 613MB of RAM) run Windows Server 2008 and SQL Server without running out of memory? My company wants to use EC2 to host our site, but we want to keep costs low during development.
I think Windows Micro (613MB RAM) instances are great for their true cloud purposes: namely on-demand access to compute power for transient activities, such as training environments or basic dev/playground/lab activities. For running an actual website, with real clients and potentially some load, you really can't go less than a small instance size. Costs aren't too bad if you buy a Small Reserved Instance, but don't go too cheap and try to use the Micro in production.