I've spent some time now looking for information regarding elasticsearch.yml configurations that make my single instance Elasticsearch (on Windows 2012 Server EC2) accessible via public ip, but everytime I uncomment one or both of following settings the only thing that changes is, calling the private ip as well results in an error.
network.publish_host: <public ip>
network.bind_host: <private ip>
Is this correct and are there any other settings that have to be defined? Shouldn't it run with the default values?
This is more of a general answer as to how networking works within EC2 instead of a specific answer to your question. But it should help inform how to configure your application.
EC2 has 1:1 NAT between a public and private IP address. Because of this, only the private IP address is visible to the instance directly.
If you are binding a service to a network interface, it would be the one with the private IP.
Some services do require knowledge of the external IP address in order to function properly. The only one I have run into is FTP in a passive configuration, likely due to the fact that it needs to open a separate socket for data transfer.
In the case of elastic search, it appears that they have a special plugin that will help configure elastic search for the aws environment: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-network.html
I had the same problem.
Installed only one instance of ES on aws EC2 and wanted to grant it public access.
On ubuntu 16.04 this is what works for me:
in /etc/elasticsearch/elasticsearch.yml add this line:
network.host: <ec2 instance private ip>
The private ip should be something like 172.x.x.x
Also do not forget allow access in security group in your aws console for port 9200 (default) and ip address from which you will be sending requests.
So difference was setting not public but private ip address from aws console..
Also note that this can be dangerous as there is not any user/password or other access control
Related
I'm having problems trying to connect to Elasticsearch (ES) on an EC2 instance from my local linux box via the EC2 instance public ip i.e. curl [PUBLIC_IP]:9200
I followed the steps in this guide: https://github.com/miztiik/elk-stack/tree/master/ElasticSearch.
My ES version is 6.8.9
Here's what's working and what's not:
On ES EC2 instance: curl localhost:9200 works
On another instance with same VPC: curl [PUBLIC_IP]:9200 works
On my local linux box: curl [PUBLIC_IP]:9200 doesn't work, however telnet [PUBLIC_IP] 9200 works i.e. it connects and gives me the escape character '^]'
My /etc/elasticsearch/elasticsearch.yml config has the following:
http.enabled: true
http.port: 9200
network.host: 0.0.0.0
http.cors.allow-origin: "*"
http.cors.enabled: true
There is only one (new) security group attached to the EC2 instance, which has the following inbound rules:
I also confirmed that the EC2 instance is in a public subnet i.e. connected to an internet gateway.
Thanks for any help.
Update
I also installed Apache httpd on the instance and rechecked everything. Here is the current state of things:
I can ping, telnet and connect to the web server (:80) from the outside.
I cannot connect to Elasticsearch (:9200) or Kibana (:5601) from the outside. All these I can however do within the VPC from another instance.
This sounds firewall related.
Check on the ECE2 security group and either modify the default Sec group or create new one and associate it with your instance.
For a test, modify your inbound as for your port as:
0.0.0.0/0 IPv4
And set network host as follows
network.host: _ec2 # if using the plugin
Otherwise
network.host: "{elastic_ip}”
If your ece2 instance doesn’t have public dns, you will have to edit your/etc/hosts file and add the IP address of your instance
network.bind_host
This specifies which network interface(s) a node should bind to in order to listen for incoming requests. A node can bind to multiple interfaces, e.g. two network cards, or a site-local address and a local address. Defaults to network.host.
network.publish_host
The publish host is the single interface that the node advertises to other nodes in the cluster, so that those nodes can connect to it. Currently an Elasticsearch node may be bound to multiple
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html
https://discuss.elastic.co/t/elasticsearch-only-accessible-from-localhost/65782/3
https://www.elastic.co/blog/running-elasticsearch-on-aws
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#describing-security-group
How do I enable remote access/request in Elasticsearch 2.0?
I had the same issue on AWS. Try using the public DNS or the private IP in lieu of the public IP to connect another ec2 instance in the same VPC.
I have a 4 node hadoop cluster on ec2. We have configured Hortonworks Hadoop (HDP version 2.4) through Ambari.
I have opened all traffic for our all four instances internally and the office external IP.
Whenever I do telnet within the cluster using internal IP:
telnet <internal_ip> 2181
It is able to connect to the specific port I have my service(zookeeper) running on.
When I use public IP of the same instance(Elastic IP) instead of internal IP, I am not able to telnet either within the cluster or from my office IP:
telnet <elastic_ip> 2181
I have already configured security group to allow all traffic. I am using Ubuntu 14.04. We are not using any other firewall except AWS security group.
Please suggest how can I connect using Elastic IP/Public IP of my instance on this port.
Please find the screenshot of Security Group of EC2:
Do you use the default VPC ?
If not, check if the VPC has an Internet Gateway, the Route table (you need a route to the Internet Gateway) and the Networks ACLs.
The Route table and Network ACLs are applied to a subnet.
The default VPC is configured to allow outside traffic, not a new VPC.
Or, the Elastic IP is linked to the same network interface ? The Elastic IP is linked to a network interface of an instance.
EDIT: you can take a look on AWS doc for a better explanation:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html
I had a running instance, and then I became unable to connect to it via http(80) and ssh(22). I tried to reboot the instance, but nothing went up. This has happened to me twice in the past month.
Why does it happen? Can I do anything to fix and/or prevent it from happening?
If I launch a new instance in same region, and it works.
Things to check when trying to connect to an Amazon EC2 instance:
Security Group: Make sure the security group allows inbound access on the desired ports (eg 80, 22) for the appropriate IP address range (eg 0.0.0.0/0). This solves the majority of problems.
Public IP Address: Check that you're using the correct Public IP address for the instance. If the instance is stopped and started, it might receive a new Public IP address (depending on how it has been configured).
VPC Configuration: Accessing an EC2 instance that is launched inside a Virtual Private Cloud (VPC) requires:
An Internet Gateway
A routing table connecting the subnet to the Internet Gateway
NACLs (Network ACLS) that permit through-traffic
If you are able to launch and connect to another instance in the same subnet, then the VPC configuration would appear to be correct.
The other thing to check would be the actual configuration of the operating system on the instance itself. Some software may be affecting the configuration so that the web server / ssh daemon is not working correctly. Of course, that is hard to determine without connecting to the instance.
If you are launching from a standard Amazon Linux AMI, ssh would work correctly anytime. The web server (port 80) would require installation and configuration of software on the instance, which is your responsibility to maintain.
I have the following setup:
- a VPC, with several subnets, and an access gateway and a NAT instance having public addresses that I can connect to,
- I create a Linux instance in a subnet of the VPC, that has NO public IP address or DNS name (I want that only the Load Balancer be known on the internet).
I want to connect to my Linux instance to install and configure software.
How do you connect to that instance? All the documentation I have seen mentions that you connect using "ec2-user#".
Since I have no public DNS, i have tried to connect from the access gateway via putty with the private DNS of my linux instance but it fails ("host does not exist").
I am obviously missing something ... in the NAT?
Thanks, Laurent
You need to have a hosts in the public subnet which you can access. Once you access this host, then you can connect to your other hosts in VPC using their private IP address.
Your instance in question has only private IP address so connecting it from your workstation is not going to work.
The host I am referring to is usually called Bastion Host. read the Tip in Scenario 2: VPC with Public and Private Subnets documentation.
Also, read first few results of this Google Search to gain overall understanding on use-cases for Bastion hosts.
I'm struggling to get EC2 and ElasticSearch up and running. Specifically I'm trying to reach my node from outside Amazon's cloud for verification purposes. I've set up the security group so that I have a "Custom TCP" rule on port 9200 and ElasticSearch is listening on that port, which I can see with netstat -l. When I curl -XGET https://localhost:9200 I get the response expected from ElasticSearch. When I curl -XGET https://publicIP:9200 from WITHIN Amazon (ie. another node that I have running) I get the response expected from ElasticSearch. When I try to do the same request from my desktop I get "no response". I can not, for the life of me, figure out why this is happening.
There are several things to check:
Accessing the public URL of an instance from inside the amazon cloud will map to its private IP. In you test above, where you specify publicIP, did you use the public IP or public domain name? Make sure to test with the IP, not the domain name.
If access to the public IP works from the same machine, try the same thing from another EC2 instance.
Finally, you may have a firewall rule on your desktop, or your work network, preventing outgoing access on port 9200.
If you are running Elasticsearch as a service, then go to /etc/elasticsearch/elasticsearch.yml and make the
network.host: "0.0.0.0"
This solution worked for me.