I ran this command to get the instance-id from a EC2 instance, how is the request processed and how does the service know what details to send back ?
wget -q -O - http://169.254.169.254/latest/meta-data/instance-id
thanks
At a high level, wget is a command that initiates an HTTP web request (pretending to be a browser) and those options tell it to spit out the resulting response to stdout (what you see).
Since the EC2 dom0 host controls the network stack as seen by your instance running in a virtual machine, EC2 can handle network traffic to 169.254.169.254 any way it wants.
In this case, EC2 knows what instance is making the request (whether it's based on your internal IP address controlled by EC2, or based on the fact that the dom0 host may be processing the request before it even gets sent across the network).
So, EC2 knows what instance is making the request and EC2 knows all the information about every instance, so EC2 can return the meta-data that is requested including the instance id.
Amazon hasn't published exactly how they have implemented this feature, but they do guarantee that it will return the correct data for the requesting instance with no chance of anybody else interfering.
You can learn more about available EC2 metadata here:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?AESDG-chapter-instancedata.html
The magic IP address 169.254.169.254 and the corresponding meta-data URLs will not work outside of an EC2 instance, unless you happen to be running on a system which is trying to emulate EC2.
You can use ec2metadata
ec2metadata --instance-id
Related
I have Windows EC2 instance I use for my public-facing C# API. The VPC(and related Internet Gateway, subnets, etc) are all default.
I've now setup an AWS ElasticSearch service using their more secure VPC Endpoint option (instead of public-facing) and I've associated it to the same subnet and vpc as my above Windows EC2 instance.
I'd like to get them to talk to each other.
Reading from https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html
It seems what you'd do is ssh tunnel / port forward traffic from localhost:9200 on the EC2 instance to the actual Elastic Search service (via that VPC endpoint).
It seems this command is where the magic happens:
ssh -i ~/.ssh/your-key.pem ec2-user#your-ec2-instance-public-ip -N -L 9200:vpc-your-amazon-es-domain.region.es.amazonaws.com:443
but that is for a Linux EC2 instance.
If I am Remote Desktopped into my Windows EC2 instance (the API), how can I make it so when I go to a browser, http://localhost:9200
will send traffic to my VPC Endpoint:
vpc-your-amazon-es-domain.region.es.amazonaws.com:443
Thanks!
Alright, so I'll answer my two questions:
First, it's actually quite easy, just RDP to your box and access the instance directly via the VPC endpoint. You don't need to do anything wacky like port forwarding using the netsh command or anything like that. Simply make sure the server (in my case my API) is on the same VPC and you're fine. I just had an error in my connection string that's why it didn't connect. To confirm, I RDP'D in and was able to hit the endpoint directly in a browser on port 80. While it's true the actual Elasticsearch runs on port 9200, you don't need to forward to localhost:9200 --> vpc:9200.
Now, regarding the second question, about hitting it locally, I think the problem is that because this service lacks a public IP address and you can't access it, that you can go through some complicated setup on AWS, or easier is just set it up to run locally for now until you are ready to use the VPC one (and thus your code will just run). Another option is to use security groups and make a publicly accessible cluster for now, and then when your code is done, search service/layer done, etc, you can start anew with a VPC/secure Elasticsearch service and that should be it.
Another thing that many mention is that it is cheaper/you have more control of things if you setup your own Elasticsearch on your local machine, and then set one up on EC2 (this is just reading blogs and seeing people mention how much frustration they had with it).
I've set up 2 instances of Windows Server 2008 on EC2. I want one to act as the database server and the other as the client. For the client app to work it needs to be able to connect to the server instance with ALL of these things:
IP address of the database instance
access through a given UDP port
server name e.g. \\MyServer
an actual physical path through to its database e.g. \\UNC\SharedFolder\MyDatabaseFolder
I'm a complete novice with EC2. Is there anyway I can set this up?
Many thanks
At least three of the four are completely possible and I have worked with similar setups. Maybe someone else knows more about the UDP bit.
IP address of the database instance
That is standard on EC2. All instances have two network interfaces, one EC2 internal and one to the outside world. For communication between instances use the internal one. Data traffic over these interfaces is free.
Access through a given UDP port
I have never tried UDP communication in EC2, but if it works you should probably keep it within a local network of your own, i.e. a virtual private cloud (VPC).
Server name e.g. \MyServer
This kind of host name lookup does not need a name server, although you certainly could run one (preferably within a VPC). If you put the server name and (internal) IP into your hosts file (%systemroot%\system32\drivers\etc\hosts) you don't need a name server, though.
An actual physical path through to its database e.g. \UNC\SharedFolder\MyDatabaseFolder
Folder sharing should work the same as with any other Windows machine, but even that should probably be kept within a VPC.
Setting up a VPC can be a little steep to start with, but the documentation is good and the hard bits are often not needed (such as VPN tunnels). Have a look at the example scenarios and follow the one best matching your needs.
I have a free Amazon EC2 instance. And I installed Apache web server on it. I have the DNS record for my domain point to the ip for the EC2 instance. I can not access to my website. Then I looked up and allow the http inbound. But I still failed to access my web? What might be the reason. Anybody gives me a clue?
Go to the AWS management console and look at the Security Group the instance is in. Then make sure you have the port open that you are trying to connect to (most likely 80). To open it to the world set the ip range to 0.0.0.0/0 and to open it to a specific ip (like only your house) set it to xxx.xxx.xxx.xxx/32.
That is almost always the reason people have problems connecting when they are new to AWS. I wrote this post, which should help get you setup.
Amazon only gives you a certain number of static ip address and the local (private) ips of each EC2 instance can change when the machine is restarted. This makes creating a stable platform where EC2 instances depend on each other ridiculously hard to use as far as I can tell.
I've search online a lot about various solutions and so far have found nothing reasonable outside of assigning an elastic ip address on ever EC2 even if its not public facing. Does anyone have any other good ideas that is actually easy to execute on?
Thanks!
See the AWS team's response to question Static local IP:
The internal IP address of EC2 instances is allocated via DHCP. On
instance shutdown, or when the DHCP lease expires, the IP address is
returned to the general EC2 DHCP pool of addresses available for other
instances.
There is no way to guarantee that you will obtain the same DHCP
address across reboots.
Edit: The answer is to use Amazon VPC. There is no downside except a trivial amount of extra setup because now you control the router. It's a world apart from plain old EC2 instance on AWS. It's so necessary in fact that VPC will be enabled for all future AWS setups by default. See this post for more information: http://www.reddit.com/r/aws/comments/1a3n0r/ec2_update_virtual_private_clouds_for_everyone/
The stock answers are:
Use AWS VPC so you have complete control over instance addressing
Use Elastic IPs, which will resolve to the instance's local address (not the public, as you'd expect) when used to communicate between EC2 instances
I stumbled upon third option. There's ec2-ssh by the Instragram folks. It's a python shell script that you install globally and lets you both query the public dns of your ec2 instances by tag name and also ssh in via tag name as well.
The documentation for it is virtually nonexistent. I've written down the steps to install below:
To install ec2-ssh:
sudo yum install python-boto (python wrapper for ec2 api)
git clone https://github.com/Instagram/ec2-ssh
In your ~/.bash_profile set your AWS access key and secret like so:
export AWS_ACCESS_KEY_ID=XYZ123
export AWS_SECRET_ACCESS_KEY=XYZ123
cd into the bin folder of the repo, there will be two files:
ec2-host and ec2-ssh
copy them to your /usr/bin or /usr/local/bin.
Now you can do awesome stuff like:
$ ec2-host ZenWorker
ec2-999-xy-999-99.compute-1.amazonaws.com
and
$ ec2-ssh ZenWorker
Connecting to ec2-999-xy-999-99.compute-1.amazonaws.com.
Note that in your regular shell scripts you can use backticks to call these global tools. I've timed these calls and they take between 0.25 and 0.5 second using an EC2 instance, so that's really the only downside. Perhaps you can live with the delay, or use the fact that public DNS only changes for an instance on reboot to work up a solution.
Note that these two programs are commandline scripts and you don't need any Python knowledge to use them. For PHP fans, or those that also want an easy way to scp files without knowing the changing public DNS, you can checkout ec2dns.
I was in the same situation once. I still dont have the expertise to solve it properly. My ugly solution was to use elb not really for load balancing but just for the endpoint.
But I think a good solution can be obtained by using aws vpc.
Here's another Ruby solution for Updating Route 53 DNS from instance on AWS. You shouldn't reference raw 3rd party system IP addresses in your applications or server configurations.
you can change Ip Address using Elastic Ip:
You Can Do Using C# Code:
var associateRequest = new AssociateAddressRequest
{
PublicIp = your Elastic Ip,
InstanceId = Your Instance Id Which You Assign
};
amazonEc2Client.AssociateAddress(associateRequest);
after That DeAssociate It.
var disAssociateRequest = new isassociateAddressRequest(publicIp.ElasticIpAddress1);
AmazonEc2Client.DisassociateAddress(your Elastic Ip);
your Public Ip Will Change
I have a three-tier Windows-based web application bundled into 3 AMIs on Amazon EC2 that I use for load testing.
An ASP.NET web application on IIS
An .NET application server
SQL Server
After I launch them, the config files of each tier needs modifying to update the IP addresses.
At the moment I am doing this manually: I connect to the webserver instance via remote desktop and modify the config file to point to the new IP of the application server instance. Then I do the same with the application server to change the IP in the connection string.
This must be a common requirement and I must be missing something obvious. There must be a better way!
I could use Elastic IP addresses, but these machines are only provisioned for a couple of hours at a time, and I would be charged for the addresses when they were NOT in use (which would be most of the time).
Is there some way of persistently naming the machines? Can I somehow get all the machines on the same network and use machine names instead of IP addresses?
I could write some nifty PowerShell script that would perform the modifications remotely. Is there an example somewhere?
I could use a dynamic IP address service. I'm not sure if this would have any negative effect on performance or availability... Are there any downsides to this approach?
I could install some sort of self-configuring service on each machine (which connects to S3? SNS? SimpleDB?) to publish/retrieve the addresses of the other machines and update the config files automatically. Is there an example somewhere?
What is best practice?
You could use Amazon Virtual Private Cloud (Amazon VPC). You have a private subnet where you can assign an IP address to an instance, but it may require launching an instance from command line to assign IP. VPC is charged the same way as EC2.