I'm struggling to get EC2 and ElasticSearch up and running. Specifically I'm trying to reach my node from outside Amazon's cloud for verification purposes. I've set up the security group so that I have a "Custom TCP" rule on port 9200 and ElasticSearch is listening on that port, which I can see with netstat -l. When I curl -XGET https://localhost:9200 I get the response expected from ElasticSearch. When I curl -XGET https://publicIP:9200 from WITHIN Amazon (ie. another node that I have running) I get the response expected from ElasticSearch. When I try to do the same request from my desktop I get "no response". I can not, for the life of me, figure out why this is happening.
There are several things to check:
Accessing the public URL of an instance from inside the amazon cloud will map to its private IP. In you test above, where you specify publicIP, did you use the public IP or public domain name? Make sure to test with the IP, not the domain name.
If access to the public IP works from the same machine, try the same thing from another EC2 instance.
Finally, you may have a firewall rule on your desktop, or your work network, preventing outgoing access on port 9200.
If you are running Elasticsearch as a service, then go to /etc/elasticsearch/elasticsearch.yml and make the
network.host: "0.0.0.0"
This solution worked for me.
Related
I followed this guide to install ELK on an EC2 instance: https://logz.io/learn/complete-guide-elk-stack/#installing-elk
Now what I'm trying to do is to view Kibana, for example, on my browser since EC2 doesn't have a GUI; it's all a CLI.
Now what should I change in the Kibana and/or Elasticsearch config files to allow this to happen? As Kibana is on port 5601, how can I see it from the browser like this?
ec2-x-xx-xx-xx.us-east-2.compute.amazonaws.com:5601
Whenever I put the above in the browser, nothing happens. I tried to change the network.host field to the public IP of the EC2 instance, but that didn't help. What am I missing here?
Updating answer based on comment thread
Looking through the guide you linked, a few things are missing:
Bind Kibana to an externally accessible port. In /etc/kibana/kibana.yml:
server.host: "<your private EC2 ip>"
Add port 5601 to your security group
The rule will look like this if you want it accessible to the world (though I'd recommend you select the "My IP" option from Source if you don't enable security):
Once you've done those things, then you, you can hit Kibana using the public IP/DNS for your EC2 instance like http://ec2-IP.aws.com:5601
I am using elastic search in my ubuntu16.04 server. When i am trying to access elasticsearch from browser by using url ip:port/_cat/indices?v. I am getting site can't be reached. After that i am change the network.host value to network.host: 0.0.0.0. After change the network.host ip the search engine not started. How can i access the elasticsearch in my browser.I changed the port also.
Thank you..
There can be many reasons for ES not being reachable. I would start with the obvious and make sure that:
ES is listening on the port: on the ES instance when you run 'curl
ip:port' you should get an answer. if not the service didn't start
well.
make sure there are no firewall rules/security groups that prevent
access from remote network.
make sure network.publish_host is configure correctly:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings
more info here: ElasticSearch instance not reachable from outside the server - Azure Windows 2012
I have Windows EC2 instance I use for my public-facing C# API. The VPC(and related Internet Gateway, subnets, etc) are all default.
I've now setup an AWS ElasticSearch service using their more secure VPC Endpoint option (instead of public-facing) and I've associated it to the same subnet and vpc as my above Windows EC2 instance.
I'd like to get them to talk to each other.
Reading from https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html
It seems what you'd do is ssh tunnel / port forward traffic from localhost:9200 on the EC2 instance to the actual Elastic Search service (via that VPC endpoint).
It seems this command is where the magic happens:
ssh -i ~/.ssh/your-key.pem ec2-user#your-ec2-instance-public-ip -N -L 9200:vpc-your-amazon-es-domain.region.es.amazonaws.com:443
but that is for a Linux EC2 instance.
If I am Remote Desktopped into my Windows EC2 instance (the API), how can I make it so when I go to a browser, http://localhost:9200
will send traffic to my VPC Endpoint:
vpc-your-amazon-es-domain.region.es.amazonaws.com:443
Thanks!
Alright, so I'll answer my two questions:
First, it's actually quite easy, just RDP to your box and access the instance directly via the VPC endpoint. You don't need to do anything wacky like port forwarding using the netsh command or anything like that. Simply make sure the server (in my case my API) is on the same VPC and you're fine. I just had an error in my connection string that's why it didn't connect. To confirm, I RDP'D in and was able to hit the endpoint directly in a browser on port 80. While it's true the actual Elasticsearch runs on port 9200, you don't need to forward to localhost:9200 --> vpc:9200.
Now, regarding the second question, about hitting it locally, I think the problem is that because this service lacks a public IP address and you can't access it, that you can go through some complicated setup on AWS, or easier is just set it up to run locally for now until you are ready to use the VPC one (and thus your code will just run). Another option is to use security groups and make a publicly accessible cluster for now, and then when your code is done, search service/layer done, etc, you can start anew with a VPC/secure Elasticsearch service and that should be it.
Another thing that many mention is that it is cheaper/you have more control of things if you setup your own Elasticsearch on your local machine, and then set one up on EC2 (this is just reading blogs and seeing people mention how much frustration they had with it).
I am running solr inside minikube for a POC. I am trying to figure out how to access the solr inside the minikube. As per my knowledge I cant access solr using my host IP, it is only accessible using the minikube IP - 192.168.99.100:8983/solr. My objective is to hit the solr server when accessing it from a remote box.
One of my team mates suggested that I can maybe use something that will forward incoming request to a local IP.
Any suggestions??
Thanks
You would need to expose the solr service using the kubectl expose command for external access.
There are four ways to expose a service for external access in k8s:
LoadBalancer service type which sets the ExternalIP automatically. This is used when there is an external non-k8s, cloud-provider's load-balancer like CGE, AWS or Azure, and this external load-balancer would provide the ExternalIP for the nginx ingress service per https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services---service-types.
ExternalIPs per https://kubernetes.io/docs/concepts/services-networking/service/#external-ips.
NodePort: In this approach, the service can be hit from outside the cluster using NodeIP:NodePort/url/of/the/service.
Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/
First you need to get URL of your Service for solr.
$ minikube service <service-name> --url
http://192.168.99.100:30000
Here, 30000 is your solr Service NodePort.
Now you need to create SSH tunnel.
For that try this
$ ssh -i ~/.minikube/machines/minikube/id_rsa docker#$(minikube ip) -L \*:30000:0.0.0.0:30000
Note: To keep SSH tunneling in Background, add & at the end of (ssh -i .....) &
Now you can access this solr Service using your Host IP address
I've spent some time now looking for information regarding elasticsearch.yml configurations that make my single instance Elasticsearch (on Windows 2012 Server EC2) accessible via public ip, but everytime I uncomment one or both of following settings the only thing that changes is, calling the private ip as well results in an error.
network.publish_host: <public ip>
network.bind_host: <private ip>
Is this correct and are there any other settings that have to be defined? Shouldn't it run with the default values?
This is more of a general answer as to how networking works within EC2 instead of a specific answer to your question. But it should help inform how to configure your application.
EC2 has 1:1 NAT between a public and private IP address. Because of this, only the private IP address is visible to the instance directly.
If you are binding a service to a network interface, it would be the one with the private IP.
Some services do require knowledge of the external IP address in order to function properly. The only one I have run into is FTP in a passive configuration, likely due to the fact that it needs to open a separate socket for data transfer.
In the case of elastic search, it appears that they have a special plugin that will help configure elastic search for the aws environment: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-network.html
I had the same problem.
Installed only one instance of ES on aws EC2 and wanted to grant it public access.
On ubuntu 16.04 this is what works for me:
in /etc/elasticsearch/elasticsearch.yml add this line:
network.host: <ec2 instance private ip>
The private ip should be something like 172.x.x.x
Also do not forget allow access in security group in your aws console for port 9200 (default) and ip address from which you will be sending requests.
So difference was setting not public but private ip address from aws console..
Also note that this can be dangerous as there is not any user/password or other access control