I have installed ElasticSearch 5.5 on EC2 (not using managed service). the EC2 has elastic ip and all incoming and outgoing rules are open.
I am able to add data and query the mapping, cluster status, templates and so on, however when i try to run a query a data of a certain index, I am getting a timeout. when sending the request within the VPC (using private IP) all is ok. I run tcpdump to see what is going on and for my surprise I didnt see any incoming request when i did a search query, all the other requests were seen in the dump. Then I have installed on a new EC2 elasticsearch cluster, and again the same behavior
what is going on?
it was a firewall issue that blocked the request
Related
I am using elastic search in my ubuntu16.04 server. When i am trying to access elasticsearch from browser by using url ip:port/_cat/indices?v. I am getting site can't be reached. After that i am change the network.host value to network.host: 0.0.0.0. After change the network.host ip the search engine not started. How can i access the elasticsearch in my browser.I changed the port also.
Thank you..
There can be many reasons for ES not being reachable. I would start with the obvious and make sure that:
ES is listening on the port: on the ES instance when you run 'curl
ip:port' you should get an answer. if not the service didn't start
well.
make sure there are no firewall rules/security groups that prevent
access from remote network.
make sure network.publish_host is configure correctly:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings
more info here: ElasticSearch instance not reachable from outside the server - Azure Windows 2012
I have Windows EC2 instance I use for my public-facing C# API. The VPC(and related Internet Gateway, subnets, etc) are all default.
I've now setup an AWS ElasticSearch service using their more secure VPC Endpoint option (instead of public-facing) and I've associated it to the same subnet and vpc as my above Windows EC2 instance.
I'd like to get them to talk to each other.
Reading from https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html
It seems what you'd do is ssh tunnel / port forward traffic from localhost:9200 on the EC2 instance to the actual Elastic Search service (via that VPC endpoint).
It seems this command is where the magic happens:
ssh -i ~/.ssh/your-key.pem ec2-user#your-ec2-instance-public-ip -N -L 9200:vpc-your-amazon-es-domain.region.es.amazonaws.com:443
but that is for a Linux EC2 instance.
If I am Remote Desktopped into my Windows EC2 instance (the API), how can I make it so when I go to a browser, http://localhost:9200
will send traffic to my VPC Endpoint:
vpc-your-amazon-es-domain.region.es.amazonaws.com:443
Thanks!
Alright, so I'll answer my two questions:
First, it's actually quite easy, just RDP to your box and access the instance directly via the VPC endpoint. You don't need to do anything wacky like port forwarding using the netsh command or anything like that. Simply make sure the server (in my case my API) is on the same VPC and you're fine. I just had an error in my connection string that's why it didn't connect. To confirm, I RDP'D in and was able to hit the endpoint directly in a browser on port 80. While it's true the actual Elasticsearch runs on port 9200, you don't need to forward to localhost:9200 --> vpc:9200.
Now, regarding the second question, about hitting it locally, I think the problem is that because this service lacks a public IP address and you can't access it, that you can go through some complicated setup on AWS, or easier is just set it up to run locally for now until you are ready to use the VPC one (and thus your code will just run). Another option is to use security groups and make a publicly accessible cluster for now, and then when your code is done, search service/layer done, etc, you can start anew with a VPC/secure Elasticsearch service and that should be it.
Another thing that many mention is that it is cheaper/you have more control of things if you setup your own Elasticsearch on your local machine, and then set one up on EC2 (this is just reading blogs and seeing people mention how much frustration they had with it).
I am trying to understand how Kibana communicates with ElasticSearch. Does this communication happen b/w the browser and the ElasticSearch server directly (Bypassing the Kibana server) or all Kibana requests go through Kibana server which then pass them on to ElasticSearch.
I am trying to figure out how to implement security around my ElasticSearch & Kibana based servers.
In Kibana3, the browser would talk to elasticsearch directly. Any protection of elasticsearch had to be done via a front-end web server, typically nginx with a bunch of proxy rules to prevent certain actions, etc.
In Kibana4, they realized that having a server would be beneficial, so your browser talks to the kibana server, which talks to elasticsearch.
The "official" way to protect elasticsearch is with "shield", which handles authentication and authorization for actions against the server.
in kibana 5 a request is sent from a web browser to kibana backend(developped using hapi js) then this request will be sent to elasticsearch using elasticsearch plugin wich is one of the kibana core plugins.
I'm trying to track down who is issuing queries to an ElasticSearch Cluster. Elastic doesn't appear to have an access log.
Is there a place where I can find out which IP is hitting the cluster?
Elasticsearch doesn't provide any security out of the box, and that is on purpose and by design.
So you have a couple solutions out there:
Don't let your ES cluster exposed to the open world, but put it behind a firewall (i.e. whitelist the hosts that can access ports 9200/9300 on your nodes)
Look into the Shield plugin for Elasticsearch in order to secure your environment.
Put an nginx server in front of your cluster to act as a reverse proxy.
Add simple basic authentication with either the elasticsearch-jetty plugin or simply the elasticsearch-http-basic plugin, which also allowws you to whitelist the client IPs that are allowed to access your cluster.
If you want to have access logs, you need either 2 or 3, but all solutions above will allow you to secure your ES environment.
For what reason would an elastic IP disassociate from a running Instance without any API calls being made?
I am experiencing an odd situation where my running AWS instance is being disassociated from the elastic IP address without any action on my part.
I tried allocating a new address and associating the instance with the new Elastic IP, but the same situation occurs where the IP address is "automatically" disassociated from the instance.
The only thing I haven't tried is stopping and restarting the instance (to move it to new hardware) but I'd rather not lose my internal IP address.
Has anyone else experience such an issue?
It seems that there is currently an issue with Elastic IP address that requires an Instance stop/start in order to ensure that the Elastic IP address "sticks"
https://forums.aws.amazon.com/thread.jspa?threadID=84952&tstart=0
We've noticed this happening to our instances also... I think the problem is a result of the internal IP changing because they are dynamic. When the internal IP changes, the elastic IP drops and you have to associate it again. It's happened to us during a reboot we initiated and it happened on an amazon reboot d