How to config Single node for Single Cluster (Standalone Cluster) ElasticSearch - elasticsearch

I installed elastic search in my local machine, I want to configure it as the only one single node in the cluster(Standalone Server). it means whenever I create a new index, it will only available to my server. It will not be accessible to other's server.
My current scenario these indexes are available to other servers (the servers are formed in a cluster), and they can make any changes to my indexes. But I don't want it.
I went through some other blogs but not getting best solution. So can you please let me know steps for same?

I ve got the answer from http://elasticsearch-users.115913.n3.nabble.com/How-to-isolate-elastic-search-node-from-other-nodes-td3977389.html.
Kimchy : You set the node to local(true), this means it will not discover other nodes using network, only within the same JVM.
in elasticsearch/bin/elasticsearch.yml file
node.local: true # disable network
Updated for ES 7.x
in elasticsearch.yml
network.host: 0.0.0.0
discovery.type: single-node
and make sure you have cluster.initial_master_nodes off
# cluster.initial_master_nodes: ["node-1", "node-2"]
credited to #Chandan.

In elasticsearch.yml
# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
index.number_of_shards: 1
index.number_of_replicas: 0
Use the same configuration in your code.
Also to isolate the node use node.local: true or discovery.zen.ping.multicast: false

Here's relevant info for ElasticSearch 5:
According to changelog, to enable local mode on ES 5 you need to add transport.type: local to your elasticsearch.yml instead of node.local: true.

If you intend to run Elasticseach on a Single Node and be able to bind it to public IP, two important settings are:
network.host: <PRIVATE IP OF HOST>
discovery.type: single-node

If you're using a network transport in your code, this won't work, as node.local gives you a LocalTransport only:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-transport.html#_local_transport
The trick then is to set
discovery.zen.ping.multicast: false
in your elasticsearch.yml which will stop your node looking for any other nodes.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#multicast
I'm not sure if this prevents other nodes from discovering yours though; I only needed this to affect a group of nodes with the same settings on the same network.

I wanted to do this without having to write/overwrite an elasticsearch.yml in my container. Here it is without a config file
Set an environment variable prior to starting elasticsearch:
discovery.type=single-node
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

In the config file, add:
network.host: 0.0.0.0 [in Network settings]
discovery.type: single-node [in Discovery and Cluster formation settings]

This solve your problem:
PUT /_all/_settings
{"index.number_of_replicas":0}
Tested with ES version 5.

All of these didn´t help me (and I sadly didn´t read the answer of bhdrkn). The thing that worked for me was to change elasticsearch´s cluster-name everytime I need to have a separate instance, where new nodes aren´t added automatically via multicast.
Just change cluster.name: {{ elasticsearch.clustername }} in elasticsearch.yml, e.g. via Ansible. This is particulary helpful, when building separate Stages like Dev, QA and Production (which is a standard usecase in enterprise-environments).
And if you´re using logstash to get your data into elasticsearch, don´t forget to put the same cluster-name into the output-section, like:
output {
elasticsearch {
cluster => "{{ elasticsearch.clustername }}"
}
}
Otherwise your "logstash-*"-index will not be build correctly...

Related

Can't start ElasticSearch on Mac

I installed elasticsearch by brew install elasticsearch and started it with brew services start elasticsearch, however, curl http://127.0.0.1:9200 shows connection refused. I checked the port: netstat -a -n | grep tcp | grep 9200 and some ipv4 is running there. Ok, so I opened /usr/local/etc/elasticsearch/elasticsearch.yml and changed the port to 9300 and also uncommented and changed: network.host: 127.0.0.1. Still shows connection refused when I do curl http://127.0.0.1:9300. The OS is MacOS High Sierra 10.13.4. If we open /usr/local/var/log/elasticsearch/elasticsearch_nikitavlasenko.log the error seems to be:
Cluster name [elasticsearch_nikitavlasenko] subdirectory exists in data paths [/usr/local/var/lib/elasticsearch/elasticsearch_nikitavlasenko]. All data under these paths must be moved up one directory to paths [/usr/local/var/lib/elasticsearch]
Did you have an older version (2.x or before) installed before? It sounds a lot like this PR to check that you're not using the old behavior when there was the node name in the path.
What I would do:
If you don't need the data any more, just remove /usr/local/var/lib/elasticsearch/elasticsearch_nikitavlasenko and start fresh.
If you need the data, you could either change path.data in your config or move the folder one level up (just like the log message says).
PS: I wouldn't use port 9300 for HTTP, because that's generally the port used for communication of the nodes in a cluster itself.
This was the result of a bug in the Homebrew formula for Elasticsearch. It was creating a directory with the node name which is no longer allowed for Elasticsearch.
The formula has been updated to remove node name from path.data and no longer create the invalid directory which should resolve this problem.
Ran into this issue some time back, Please add a minimal Elastic config file. for me it looks like below
http.port: 9200
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
path.data: /usr/local/var/elasticsearch/
path.logs: /usr/local/var/log/elasticsearch/
# Set both 'bind_host' and 'publish_host':
network.host: 127.0.0.1
# 1. Disable multicast discovery (enabled by default):
discovery.zen.ping.multicast.enabled: false
script.engine.groovy.inline.aggs: on
I think I wasn't having below config which caused the issue:
network.host: 127.0.0.1
Please check if its there in your config? Also properly set your data and logs folder path.
Let me know if you face any issue and have questions on these configs.

Configuring elastic search not to be localhost

After installing Elasticsearch 5.6.3 and setting Nodename to the server name. I tried to browse to Elasticsearch using IP:9200 but it didn't work. If I browse to localhost:9200 it works. Where do I go to change th default behaviour of Localhost. Since I want to open this up to other external servers so the loop back address of localhost isn't any good.
After installing Kibana 5.6.3, the same is obviously true here as well. Starting the kibana server with the ip fails, but with localhost doesn't.
At this point I have no indexes, I just want to prove Elasticsearch can be reached beyond localhost.
Thanks
Bill
You can configure your IP with the "network.host" setting in 'elasticsearch.yml' and 'kibana.yml' in your config directory.
Here is some link to the Elasticsearch doc to config yours :)
Configuring Elasticsearch
Important Settings
For a quick start development configuration the following settings can be placed into 'elasticsearch.yml':
network.host e.g.
network.host: 192.168.178.49
cluster.initial_master_nodes e.g.
cluster.initial_master_nodes: ["node_1"]
You can also define a cluster name:
cluster.name: my-application
Start it with the node name (example for Windows)
C:\InstallFolder\elasticsearch-7.10.0>C:\InstallFolder\elasticsearch-7.10.0\bin\elasticsearch.bat -Enode.name=node_1
Go to your browser and open http://192.168.178.49:9200 (replace with your IP). It shows a JSON result. The localhost:9200 will no longer work.
This config should not be used for production environments. See the official docs.
In general when starting from a command prompt it shows any errors when something fails. These are very helpful.

ElasticSearch VM clone - master_not_found_exception, found existing node with the same id but is a different node instance

Here is my setup:
Two instances of Ubuntu 16.04. Second one is clone made from the first one. ElasticSearch is installed only on Guest (Ubuntu) OSes. Configuration has been adjusted after cloning the VM.
I am running with bridged network in VirtualBox - each instance got its IP from the router. Windows (host) firewall is configured appropriately. All machines can ping each other. Ping, Netstat and nmap testing shows that ports 9200 and 9300 are OPEN (tested "remote" hosts also).
ElasticSearch service is running appropriately. I can "curl -XGET" both locally and remotely and get the correct results.
The problem is that the ES from the second machine is not joining the cluster.
Here are the configuration files:
First one:
cluster.name: p4g4n_cluster
node.name: master
node.master: true
network.host: 192.168.0.12
discovery.zen.ping.unicast.hosts: ["192.168.0.12", "192.168.0.17"]
Second one:
cluster.name: p4g4n_cluster
node.name: node1
node.master: false
network.host: 192.168.0.17
discovery.zen.ping.unicast.hosts: ["192.168.0.12", "192.168.0.17"]
if I try curl -XGET 192.168.0.17:9200/_cluster/health I will get master_not_discovered_exception. And if I try basic GET request, I will see that the node1 has _na_ for the cluster_uuid" property, while on first machine - *master*cluster_uuid` is present.
Version of ElasticSearch running is: 5.4.0 and
Version of Lucene is: 6.5.0
Can anyone help me with what needs to happen in order for node1 to see and join the cluster?
I was able to solve this issue.
Digging through the logs showed that this was not a network configuration issue.
Since I first configured the entire ELK stack on one machine and then cloned it, the ES and logstash were already running and pumping syslog logs into the elastic.
Because of this, the cloned machine had the same data folder as the existing one. As it turned out, the node UUID is embedded in the data folder and the solution was to delete the data folder on the cloned VM.
The error that I found in logs was: found existing node {xxx} with the same id but is a different node instance ... So there was an obvious conflict.
I found this github ES issue and this SO answer that dealt with the same issue.
You can try to add network.bind_host: 0.0.0.0 in both servers

How to configure two app server nodes to connect to same elasticsearch cluster?

I have one code base to connect elastic search (localhost:9200) for the full-text-search feature. We deployed this code on two different machines (m1 & m2) under load balancing server. In this case, how to configure ES in 2 different machines to connect ES and index should reflect both sides.
I am using Elasticsearch v 5.1.2
Machine 1
cluster.name: production
node.name: database
Machine 2
cluster.name: production
node.name: app
Above setting worked on ES v 1.7.1
**Question?
What configuration should I do to make it work on ES v5.1.2?
Please help me to solve this issue.
Thanks in advance
I'm assuming these nodes aren't a part of same cluster.
Try http://MACHINE_1_IP:9200/_cat/nodes?v and check if all nodes are listed as part of cluster.
If they are not - just a quick guess, have you looked at network.host setting ? It binds to local loop by default ( That maybe something introduced in 2 + )
This can be solved by using network-module setting (ref).
Update the elasticsearch.yml on both app server by keeping same cluster name and different node name
EX :
Server_1
update the elasticseach.yml
cluster.name: Production
node.name: APP
network.host: [server_1_IP, _local_]
discovery.zen.ping.unicast.hosts: [server_1_IP, server_2_IP]
On Server_2
update the elasticseach.yml
cluster.name: Production
node.name: DB
network.host: [server_2_IP, _local_]
discovery.zen.ping.unicast.hosts: [server_1_IP, server_2_IP]

how to run multiple instances of elasticsearch on one host

I have several machines each with 128 GB of ram, each host is running a single instance of Elasticsearch.
I would like to run another data node on each host and allocate around 30 GB to the jvm heap.
I know I have to create a separate config file .yml and data directory..etc. My question is do I need to modify the service wrapper so that each node will be started/ stopped seperatly?
I am running ES version 1.3 on Centos 6.5
thank you
You need to prepare two elasticsearch.yml config files to configure settings accordingly and specify these files when startup up the two nodes.
bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.1.yml
bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.2.yml
At least the following should be set differently for the two nodes:
http.port
transport.tcp.port
path_data
path_logs
path_pid
node.name
The following needs to point to the other in both files to allow the nodes to find each other:
discovery.zen.ping.unicast.hosts: '127.0.0.1:9302'
EDIT: the property is now deprecated, look at : https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-settings.html
See this blog and this discussion
elasicsearch.yml-1
cluster.name: test
node.name: node-1
path.data: /Users/musab/Desktop/elasticsearch/data
path.logs: /Users/musab/Desktop/elasticsearch/logs
node.max_local_storage_nodes: 4
elasicsearch.yml-2
cluster.name: test
node.name: node-1
path.data: /Users/musab/Desktop/elasticsearch/data
path.logs: /Users/musab/Desktop/elasticsearch/logs
node.max_local_storage_nodes: 4

Resources