I have a couple of ec2 instances sitting behind a load balancer. How can I determine which requests went to a particular instance?
Is there a log? Somewhere on the console I can look at?
Dont know of an direct way on the console. You could either activate ELB access logs and look at the backend:port field in the logs which gives the instaces ip (docs here). Or you could have your application running on your instances add something to the headers to identify themselves in their responses.
Related
So i have setup a laravel application and hosted on a docker which in turned hosted using AWS ECS Cluster running behind ALB.
So far i have the application up and running as expected, everything runs just the way it is (e.g. Sessions are stored in memcached and working, static assets are in S3 bucket, etc).
Right now i just have 1 problem with stability and i am not quiet sure where exactly the problem is. When i hit my URL / website, sometimes (randomly) it returns 502/503 HTTP error. When this happen i have to wait for about a minute or 2 before the app can return 200 HTTP code.
Here's a result of doing tail on my docker (i.e. nginx log)
At this point i am totally lost and not sure where else i should check. I've tried the following:
Run it locally, with the same docker / nginx >> works just fine.
Run it without ALB (i.e. Using just 1 EC2) >> having similar problem.
Run it using ALB on 2 different EC2 type (i.e. t2.small and micro) >> both having similar problem.
Run it using ALB on just 1 EC2 >> having similar problem.
According to your logs, ngjnx is answering 401 Unauthorized to the ALB health check request. You have to answer 200 OK in / endpoint or configure a different one like /ping in your ALB target group.
To check the health of your targets using the console
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
On the navigation pane, under LOAD BALANCING, choose Target Groups.
Select the target group.
On the Targets tab, the Status column indicates the status of each target.
If the status is any value other than Healthy, view the tooltip for more information.
More info: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html
I have had a similar issue in the past for one of a couple of possible reasons;
Health checks configured for the ALB, e.g. the ALB is waiting for the configured number of checks to go green (e.g. every 30 seconds hit an endpoint and wait for a 200 for 4/5 times. During the "unhealthy phase" the instance may be designated offline. This usually happens most often immediately after a restart or deployment or if an instance goes unhealthy.
DNS within NGINX. If the DNS records of the downstream service that NGINX is proxying have changed it might be that NGINX has cached (either according to the TTL or for much longer depending on your configuration) the old record and is therefore unable to connect to the downstream.
To help fully debug, it might be worth determining whether the 502/503 is coming from the ALB or from NGINX. You might be able to determine this from the access log of the ALB or the /var/log/nginx/access|error.log in the container.
It may also help to check, was there a response body on the response?
I was looking at Mesos + Marathon to manage Docker containers.
What we're trying to achieve is a way of getting an external DNS entry (test.example.com) to point to a specific set of docker containers.
The DNS entry for test.example.com points to a load balancer which translate and send the connection to one of our backend servers app.
To do this I looked into Mesos-dns. With mesos-dns I can get DNS name for each container and can resolve DNS with container IP but couldn't find out way to load balance between set of servers.
Can someone confirm if Mesos-dns provides load balancing? If yes, how can I achieve load balancing with it?
Do I need to use some other solutions like HAProxy or Bamboo to achieve this?
Thanks!!
Sumit
Yes with Mesos-DNS you can do load balancing, see for example the respective HTTP API endpoints, but it's really not recommended in the context of DC/OS: see the internal (Minuteman) and external (Marathon-lb, HAProxy-based) load balancing and service discover options in the docs.
I have an aws Elastic Load Balancer. Sometimes Elastic Load Balancer works sometimse not. Soemetimes I am able to hit the app sometimes it gives me the blank page .
Why its happening so
You should check ping path and ping port on ELB, whether you get response. It seems that ELB brings the instances down as failed healthcheck.
AWS Application Load Balancers are notoriously susceptible to DDoS attacks, a simple SYN flood will bring down a load balancer, and what's worse is that you won't know that it's down, because the AWS Dashboard doesn't expose anything about the load balancer other than some basic HTTP level metrics.
I have a free Amazon EC2 instance. And I installed Apache web server on it. I have the DNS record for my domain point to the ip for the EC2 instance. I can not access to my website. Then I looked up and allow the http inbound. But I still failed to access my web? What might be the reason. Anybody gives me a clue?
Go to the AWS management console and look at the Security Group the instance is in. Then make sure you have the port open that you are trying to connect to (most likely 80). To open it to the world set the ip range to 0.0.0.0/0 and to open it to a specific ip (like only your house) set it to xxx.xxx.xxx.xxx/32.
That is almost always the reason people have problems connecting when they are new to AWS. I wrote this post, which should help get you setup.
We usually blacklist IPs address with iptables. But in Amazon EC2, if a connection goes through the Elastic Load Balancer, the remote address will be replaced by the load balancer's address, rendering iptables useless. In the case for HTTP, apparently the only way to find out the real remote address is to look at the HTTP header HTTP_X_FORWARDED_FOR. To me, blocking IPs at the web application level is not an effective way.
What is the best practice to defend against DoS attack in this scenario?
In this article, someone suggested that we can replace Elastic Load Balancer with HAProxy. However, there are certain disadvantages in doing this, and I'm trying to see if there is any better alternatives.
I think you have described all the current options. You may want to chime in on some of the AWS forum threads to vote for a solution - the Amazon engineers and management are open to suggestions for ELB improvements.
If you deploy your ELB and instances using VPC instead of EC2-classic, you can use Security Groups and Network ACLs to restrict access to the ELB.
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/USVPC_ApplySG.html
It's common to run an application server behind a reverse proxy. Your reverse proxy is layer you can use to add DoS protection before traffic gets to your application server. For Nginx, you can look at the rate limiting module as something that could help.
You could set up an EC2 host and run haproxy there by yourself (that's what Amazon is using anyways!). Then you can apply your iptables-filters on that system.
Here's a tool I made for those looking to use Fail2Ban on aws with apache, ELB, and ACL: https://github.com/anthonymartin/aws-acl-fail2ban