What is your health check settings for elastic load balancer? I am not really well into this as my goal is to get the good settings to put where the ELB immediately failover the traffic when my 1st ec2 instance is down to the 2nd ec2 instance. Can anyone mind to share their configuration and knowledge?
Thanks.
James
Health check settings in ELB are important, but usually not that important.
1) ELB doesn't support active/passive application instances - only active/active.
2) If an application stops accepting connections or slows dramatically, load will automatically shift to the available / faster instances. This happens without the help of health checks.
3) Health checks prevent ELB from having to try to send a request to an instance in order to find out it is not well. This is good because a request to an unhealthy back end can sacrifice the request (an error will be sent to the client).
4) If your health check settings are too sensitive (such as using a 1 second timeout when some percent of your requests take longer than that) then it can pull instances out of service too easily. Too much of this and your site will appear to be down from time to time.
If you are trying a scenario with multiple availability zones and only one back-end in each zone, then the health checks are more important. If there are NO healthy back-ends in a zone, ELB will try to forward requests to another zone that has at least one healthy instance. In this case, the frequency of health checks determines the failover time, so you'll want faster checks.
Related
My main goal is to avoid a big delay in the health status updating to "Critical" when I can predict my service will go down.
I would combine this with the http health check already in place.
Considered solutions:
I have tried a TTL check, but this introduces the burden of converting the service to constantly send it's current status.
Using a TTL check with a really high ttl + sending "healthy" once on restart came to mind, but if this initial request fails the service stays unhealthy way too long.
Reducing the interval of my http health-check would mitigate the problem a bit, but also creates a bigger overhead.
If you can predict the service will go down, you should consider putting it into maintenance mode. This will remove it from DNS and API results immediately. Here is the link to the documentation on how to put service into the maintenance mode.
Health checks will always have a delay, since they are executed periodically and are designed to monitor service for unexpected downtime. Best way to have minimal impact on users, if you know service is going down due to update/upgrade/reboot/decomissioning, is to remove it before doing any work on it.
I have a problem with autoscaling with AWS Application Load Balancer.
I'm running my Jmeter tests and discovered that whenever new instance is added to autoscaling group (that is when it becomes healthy and ALB starts to route traffic to it), then for some short period of time Load Balancer forwards less requests to targets and a lot of requests are apparently stuck at Load Balancer itself.
I'm attaching 3 images that show this issue. CPU of JVM of one of instances drops and than goes back to normal, some requests are hanging for more than 30 sec. Number of requests per target drops and then goes back to trend. (see attached pictures)
I'm using sticky sessions with 3 minutes validity period.
Does any one knows what may cause this temporary "choking" when new instance is added?
It is quite crucial to our user experience. Can't actually understand why adding new instance can have such adverse effect on traffic routing.
Issue is fully reproducible.
I have a really simple setup: An azure load balancer for http(s) traffic, two application servers running windows and one database, which also contains session data.
The goal is being able to reboot or update the software on the servers, without a single request being dropped. The problem is that the health probe will do a test every 5 seconds and needs to fail 2 times in a row. This means when I kill the application server, a lot of requests during those 10 seconds will time out. How can I avoid this?
I have already tried running the health probe on a different port, then denying all traffic to the different port, using windows firewall. Load balancer will think the application is down on that node, and therefore no longer send new traffic to that specific node. However... Azure LB does hash-based load balancing. So the traffic which was already going to the now killed node, will keep going there for a few seconds!
First of all, could you give us additional details: is your database load balanced as well ? Are you performing read and write on this database or only read ?
For your information, you have the possibility to change Azure Load Balancer distribution mode, please refer to this article for details: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-distribution-mode
I would suggest you to disable the server you are updating at load balancer level. Wait a couple of minutes (depending of your application) before starting your updates. This should "purge" your endpoint. When update is done, update your load balancer again and put back the server in it.
Cloud concept is infrastructure as code: this could be easily scripted and included in you deployment / update procedure.
Another solution would be to use Traffic Manager. It could give you additional option to manage your endpoints (It might be a bit oversized for 2 VM / endpoints).
Last solution is to migrate to a PaaS solution where all this kind of features are already available (Deployment Slot).
Hoping this will help.
Best regards
I've been looking at high availability solutions such as heartbeat, and keepalived to failover when an haproxy load balancer goes down. I realised that although we would like high availability it's not really a requirement at this point in time to do it to the extent of the expenditure on having 2 load balancer instances running at any one time so that we get instant failover (particularly as one lb is going to be redundant in our setup).
My alternate solution is to fire up a new load balancer EC2 instance from an AMI if the current load balancer has stopped working and associate it to the elastic ip that our domain name points to. This should ensure that downtime is limited to the time it takes to fire up the new instance and associate the elastic ip, which given our current circumstance seems like a reasonably cost effective solution to high availability, particularly as we can easily do it multi-av zone. I am looking to do this using the following steps:
Prepare an AMI of the load balancer
Fire up a single ec2 instance acting as the load balancer and assign the Elastic IP to it
Have a micro server ping the current load balancer at regular intervals (we always have an extra micro server running anyway)
If the ping times out, fire up a new EC2 instance using the load balancer AMI
Associate the elastic ip to the new instance
Shut down the old load balancer instance
Repeat step 3 onwards with the new instance
I know how to run the commands in my script to start up and shut down EC2 instances, associate the elastic IP address to an instance, and ping the server.
My question is what would be a suitable ping here? Would a standard ping suffice at regular intervals, and what would be a good interval? Or is this a rather simplistic approach and there is a smarter health check that I should be doing?
Also if anyone foresees any problems with this approach please feel free to comment
I understand exactly where you're coming from, my company is in the same position. We care about having a highly available fault tolerant system however the overhead cost simply isn't viable for the traffic we get.
One problem I have with your solution is that you're assuming the micro instance and load balancer wont both die at the same time. With my experience with amazon I can tell you it's defiantly possible that this could happen, however unlikely, its possible that whatever causes your load balancer to die also takes down the micro instance.
Another potential problem is you also assume that you will always be able to start another replacement instance during downtime. This is simply not the case, take for example an outage amazon had in their us-east-1 region a few days ago. A power outage caused one of their zones to loose power. When they restored power and began to recover the instances their API's were not working properly because of the sheer load. During this time it took almost 1 hour before they were available. If an outage like this knocks out your load balancer and you're unable to start another you'll be down.
That being said. I find the ELB's provided by amazon are a better solution for me. I'm not sure what the reasoning is behind using HAProxy but I recommend investigating the ELB's as they will allow you to do things such as auto-scaling etc.
For each ELB you create amazon creates one load balancer in each zone that has an instance registered. These are still vulnerable to certain problems during severe outages at amazon like the one described above. For example during this downtime I could not add new instances to the load balancers but my current instances ( the ones not affected by the power outage ) were still serving requests.
UPDATE 2013-09-30
Recently we've changed our infrastructure to use a combination of ELB and HAProxy. I find that ELB gives the best availability but the fact that it uses DNS load balancing doesn't work well for my application. So our setup is ELB in front of a 2 node HAProxy cluster. Using this tool HAProxyCloud I created for AWS I can easily add auto scaling groups to the HAProxy servers.
I know this is a little old, but the solution you suggest is overcomplicated, there's a much simpler method that does exactly what you're trying to accomplish...
Just put your HAProxy machine, with your custom AMI in an auto-scaling group with a minimum AND maximum of 1 instance. That way when your instance goes down the ASG will bring it right back up, EIP and all. No external monitoring necessary, same if not faster response to downed instances.
I see that you can specify what availability zones an ELB serves. Right now, we're only setting a single EC2 instance up in a single AZ. But someday, we'll have multiple across AZ's for redundancy. Is there any reason I shouldn't just set up the ELB to serve two now, so I don't have to change it later?
Does it cost more to run your load balancer in multiple availability zones? Are there any other drawbacks to running an ELB in multiple AZ's?
No, it doesn't cost more to run load balancer in multi-az. Load Balancer costs are fix for every region, it does not vary for availability zones. Its better to have ELB right now, because when you need will increase and you add more instances to your account, you don't have to change URL in your application. ELB provides a seperate URL and in the background you can add more instances to it without any change to URL. If you provide DNS of instance right now in your application and in future, your plan to expand and have ELB, you need to change URL in application i.e., URL where your clients should connect.