I'm testing a Hadoop cluster with juju and Amazon EC2, and I would like to know how could I stop a cluster and then start it again, manteining the cluster configuration.
The problem is that after start the instances again the public addresses changes, and the juju-status command shows the machines are down.
The problem is that after start the instances again the public addresses changes
Above is true for EC2 instance. Its Public IP address changes when it is stopped and then started.
To avoid this, you have 2 options:
use Elastic IP. You can attach an EIP to you instance so that the instance will have the same IP address accross the start/stop cycles. Caveat: you can get only 5 EIPs per account.
To get rid of above limitation of 5 EIPs per account, you can setup your cluster in side a VPC where all your instances will have a private IP address and it will remain same across start/stop cycles. But you have to understand how VPC works in order to use that. Please read this.
Related
I have created an auto scaling group with EC2 machines. These EC2 machines have installed a webserver. The EC2 machines have a public IP adress and a load balancer in front of them. When querying the load balancer, I get a success message.
When creating the same scenario with no public EC2 machines for the EC2 machines, I get bad gateway 502. When I check the reachable analyzer, the ec2 machine is being reached.
Why is this the case? Is the public lb in a different network, so that I need an extra route?
I found my error. The webserver was not running, because in the user data I was trying to reach dockerhub and it could not reach it because my Ec2 machine is in a private network and that makes sense, because the reachable analyzer showed, that everything is right.
Yes, I read an article by Eric Hammond here where he mentions that the private IP would also change when restarting. A few months ago, when I first got an AWS cluster up for hadoop, I used the internal IP to configure /etc/hosts and the internal IP wouldn't change (even when the instance is stopped, i can see the internal IP).
To replicate this cluster as part of our corporate account, I created a few AMIs and used those to launch the instances. Now, the IPs are changing each time the machine is restarted.
On checking the machines that did not have the IP change, there doesn't seem to be anything special about them. They are the same simple EBS backed instances with volumes. Hmm, so what's the difference between them?
Check whether your EC2 instance is inside a VPC or not.
Instances inside VPC will retain their private IP addresses when stopped and restarted. But instances outside VPC (ie. EC2-Classic) will change their private IP address when stopped and restarted.
Unfortunately, it's not possible to move an EC2 instance from EC2-Classic to EC2-VPC. However, in many cases, you can create an AMI image of the instance and launch a new instance from the AMI inside the VPC.
I have a EC2 micro instance(ubuntu) running with tomcat and apache installed .
I am confused what happens if the system crashes , or shuts down. or If i stop it.
Will i have to install and configure all the packages(tomcat,apache) again on the instance , or will the instance on restart or reboot will have all these packages in the pre-stop state?
I know micro instance uses EBS for storage. And I assume on stop it maintains the state (i.e. it is not deleted). is it?
So on restart only the IP address changes right?
Or do i need to configure the instance again.
You are pretty much spot on with all your assumptions:
All EBS backed instances retain their installation state and configuration, which under normal circumstances shouldn't ever have any parts located on the (often called ephemeral) instance store volume(s), if any (see Root Device Storage Concepts for a detailed explanation of the differences between EBS and instance store backed AMIs).
Given the Amazon EC2 Instance Type t1.micro is EBS storage only indeed, this is guaranteed in your case.
Interestingly EBS storage only also applies to the newest regular instance types m3.xlarge and m3.2xlarge btw., so AWS might be slowly moving away from instance storage eventually.
Likewise, on restart only the IP address changes, be it a default public one or an Elastic IP address, see e.g. the FAQ Do I need one Elastic IP address for every instance that I have running?:
No. You do not need an Elastic IP address for all your instances. By
default, every instance comes with a private IP address and an
internet routable public IP address. The private address is associated
exclusively with the instance and is only returned to Amazon EC2 when
the instance is stopped or terminated. The public address is
associated exclusively with the instance until it is stopped,
terminated or replaced with an Elastic IP address. [...] [emphasis mine]
Please note that EC2 instances started within an Amazon VPC slightly differ here, insofar they don't have get a default public IP address and do retain an Elastic IP address (if any) throughout a stop/start cycle.
Is there any way to make new instances added to an autoscaling group associate with an elastic IP? I have a use case where the instances in my autoscale group need to be whitelisted on remote servers, so they need to have predictable IPs.
I realize there are ways to do this programmatically using the API, but I'm wondering if there's any other way. It seems like CloudFormation may be able to do this.
You can associate an Elastic IP to ASG instances using manual or scripted API calls just as you would any other instance -- however, there is no automated way to do this. ASG instances are designed to be ephemeral/disposable, and Elastic IP association goes against this philosophy.
To solve your problem re: whitelisting, you have a few options:
If the system that requires predictable source IPs is on EC2 and under your control, you can disable IP restrictions and use EC2 security groups to secure traffic instead
If the system is not under your control, you can set up a proxy server with an Elastic IP and have your ASG instances use the proxy for outbound traffic
You can use http://aws.amazon.com/vpc/ to gain complete control over instance addressing, including network egress IPs -- though this can be time consuming
There are 3 approaches I could find to doing this. Cloud Formation will just automate it but you need to understand what's going on first.
1.-As #gabrtv mentioned use VPC, this lends itself to two options.
1.1-Within a VPC use a NAT Gateway to route all traffic in and out of the Gateway. The Gateway will have an Elastic IP and internet traffic then whitelist the NAT Gateway on your server side. Look for NAT gateway on AWS documentation.
1.2-Create a Virtual Private Gateway/VPN connection to your backend servers in your datacenter and route traffic through that.
1.2.a-Create your instances within a DEDICATED private subnet.
1.2.b-Whitelist the entire subnet on your side, any request from that subnet will be allowed in.
1.2.c Make sure your routes in the Subnet are correct.
(I'm skipping 2 on purpose since that is 1.2)
3.-The LAZY way:
Utilize AWS Opsworks to do two things:
1st: Allocate a RESOURCE Pool of Elastic IPs.
2nd: Start LOAD instances on demand and AUTO assign them one elastic ip from the Pool.
For the second part you will need to have the 24/7 instances be your minimum and the Load instances be your MAX. AWS Opsworks now allows Cloud Watch alarms to trigger instance startup so it is very similar to ASG.
The only disadvantage of Opsworks is that instances aren't terminated but stopped instead when the load goes down and that you must "create" instances beforehand. Also you depend on Chef solo to initiate your instances but is the only way to get auto assigning EIPs to your newly created instances that I could find.
Cheers!
I want to use the Amazon Web Service free micro-instance for my different projects for testing and personal purpose. But I required some static-public IP on which I can run my server.
Is that possible? From where I can buy just IP and use it with my AWS?
EC2 Elastic IP Addresses
Elastic IPs are tied to an account, not an instance.
You need to look at AWS VPC for this.
Whilst VPC is free outside of the usual instance pricing, it doesn't work with Micro instances (the cheapest ones).
When not using VPC, you're assigned IP addresses through DHCP. When the DHCP lease expires, or you restart, your IP is released back to the pool.
VPC lets you use private IP addressing, you can use it with Elastic IPs and is much easier to integrate with a physical infrastructure setup.
If you're only testing/investigating AWS and have little or no budget to use anything other than a Micro instance, I'd just suck it up and deal with the changing of IPs.
If you've got a budget that lets you use instances other than Micro, then go for VPC.
Also, if you're doing more than testing/investigating I'd recommend starting with VPC straight away as trying to migrate from a non VPC to a VPC infrastructure is a massive PITA.
For every AWS account, 5 free elastic ips are provided. You have to just allocate them to required instance. But make sure that the allocate address(newly created elastic ip) in in use, because you will billed if the Elastic ip is not in use.
Looks like they have configured ARP statically so you can only use the IP address on an instance that was bound to that instance through the EC2 management console.
I just configured one of my instances to use a static IP address other than the one assigned through the management console and rebooted the instance.
I'm still receiving ARP responses on the old address but not receiving ARP responses on the new address at all.
Unfortunately for me, I have a not responding instance (NFS File Server) stuck in a stopping state while I attempt to terminate it.
The IP Address bound to that instance cannot be re-assigned to a replacement instance so now I have to reconfigure
On the whole pricing delima: When you come to think of it, there is a limited amount of static IPs so there must some pricing (supply and demand). This pricing is two fold: 1) for upto a limited number (5 per account) you don't have to pay. 2) if you created one you need to use it if you don't you'll be billed (to prevent every user to get 5 static IPs)