ElasticSearch is swapping - elasticsearch

I have launch my new site today and have problems with the SWAP of ElasticSearch. I don't know how to fix it because I have give enought heap memory for each of the 2 nodes, but somehow, there are problems.
I am attaching the previews from HQ. Can someone help me?
---- config /etc/elasticsearch.yml ---
cluster.name: xxx
node.name: xxx
node.data: true
node.master: true
bootstrap.mlockall: true
index.translog.flush_threshold_ops: 50000
index.merge.policy.merge_factor: 5
index.merge.policy.segments_per_tier: 10
# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 1000
threadpool.bulk.queue_size: 30000
# Index pool
threadpool.index.type: fixed
threadpool.index.size: 1000
threadpool.index.queue_size: 10000
index.cache.query.enable: true
indices.cache.query.size: 15%
index.cache.field.expire: 1h
indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire : 1h
indices.cache.filter.size: 15%
index.store.type: mmapfs
transport.tcp.compress: true;
network.bind_host: xxxx
network.publish_host: xxxx
network.host: xxxx
discovery.zen.ping.unicast.hosts: ["xxxx"]
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 10s
transport.tcp.port: 9300
http.port: 9200
http.max_content_length: 500mb
index.routing.allocation.disable_allocation: false
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms
script.engine.groovy.inline.aggs: on
script.inline: on
script.indexed: on
index.max_result_window: 40000
--- config /etc/default/elasticsearch -----
ES_HEAP_SIZE=5g
---- JAVA process -----
/usr/bin/java -Xms5g -Xmx5g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
---- node info ----
{"cluster_name":"xxxxx_search","nodes":{"Tze_hXZ2SwqQIvg4YWOcMg":{"timestamp":1460563870399,"name":"xxxxx-slave-1","transport_address":"xxxx:9301","host":"xxxxx","ip":["xxxxx:9301","NONE"],"jvm":{"timestamp":1460563870399,"uptime_in_millis":17908517,"mem":{"heap_used_in_bytes":2679603328,"heap_used_percent":50,"heap_committed_in_bytes":5333843968,"heap_max_in_bytes":5333843968,"non_heap_used_in_bytes":105721416,"non_heap_committed_in_bytes":108711936,"pools":{"young":{"used_in_bytes":15540664,"max_in_bytes":279183360,"peak_used_in_bytes":279183360,"peak_max_in_bytes":279183360},"survivor":{"used_in_bytes":2730400,"max_in_bytes":34865152,"peak_used_in_bytes":34865152,"peak_max_in_bytes":34865152},"old":{"used_in_bytes":2661332264,"max_in_bytes":5019795456,"peak_used_in_bytes":3813217632,"peak_max_in_bytes":5019795456}}},"threads":{"count":56,"peak_count":63},"gc":{"collectors":{"young":{"collection_count":1170,"collection_time_in_millis":38743},"old":{"collection_count":4,"collection_time_in_millis":220}}},"buffer_pools":{"direct":{"count":61,"used_in_bytes":15744627,"total_capacity_in_bytes":15744627},"mapped":{"count":85,"used_in_bytes":4319218096,"total_capacity_in_bytes":4319218096}}}},"DrG535FuQKygzKlSCAWwLw":{"timestamp":1460563870399,"name":"xxxxx-master-1","transport_address":"xxxxx:9300","host":"xxxx","ip":["xxxxx:9300","NONE"],"attributes":{"master":"true"},"jvm":{"timestamp":1460563870399,"uptime_in_millis":17912689,"mem":{"heap_used_in_bytes":2315059272,"heap_used_percent":43,"heap_committed_in_bytes":5333843968,"heap_max_in_bytes":5333843968,"non_heap_used_in_bytes":118353088,"non_heap_committed_in_bytes":121683968,"pools":{"young":{"used_in_bytes":172840784,"max_in_bytes":279183360,"peak_used_in_bytes":279183360,"peak_max_in_bytes":279183360},"survivor":{"used_in_bytes":2480072,"max_in_bytes":34865152,"peak_used_in_bytes":34865152,"peak_max_in_bytes":34865152},"old":{"used_in_bytes":2139738416,"max_in_bytes":5019795456,"peak_used_in_bytes":3826731840,"peak_max_in_bytes":5019795456}}},"threads":{"count":59,"peak_count":71},"gc":{"collectors":{"young":{"collection_count":1368,"collection_time_in_millis":47571},"old":{"collection_count":5,"collection_time_in_millis":270}}},"buffer_pools":{"direct":{"count":71,"used_in_bytes":24539898,"total_capacity_in_bytes":24539898},"mapped":{"count":84,"used_in_bytes":4318926707,"total_capacity_in_bytes":4318926707}},"classes":{"current_loaded_count":9552,"total_loaded_count":9695,"total_unloaded_count":143}}}}}
---- node process ---
{"cluster_name":"xxxx_search","nodes":{"Tze_hXZ2SwqQIvg4YWOcMg":{"name":"xxxx-slave-1","transport_address":"xxxx:9301","host":"xxxx","ip":"xxxx","version":"2.3.1","build":"bd98092","http_address":"xxxx:9201","process":{"refresh_interval_in_millis":1000,"id":25686,"mlockall":false}},"DrG535FuQKygzKlSCAWwLw":{"name":"xxxx-master-1","transport_address":"xxxx:9300","host":"xxxx","ip":"xxxx","version":"2.3.1","build":"bd98092","http_address":"xxxx:9200","attributes":{"master":"true"},"process":{"refresh_interval_in_millis":1000,"id":25587,"mlockall":false}}}}
Thanks!

Related

How do I connect two nodes in elasticsearch?

Now I have two nodes(192.168.72.129, 192.168.72.130)
It's the setting in config/elasticsearch.yml
======node-1======
cluster.name: cluster-es
node.name: node-1
network.host: 0.0.0.0
node.master: true
node.data: true
http.port: 9200
http.cors.allow-origin: "*"
http.cors.enabled: true
transport.port: 9300
http.max_content_length: 200mb
cluster.initial_master_nodes: ["node-1"]
discovery.seed_hosts: ["192.168.72.129","192.168.72.130"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
cluster.routing.allocation.cluster_concurrent_rebalance: 16
cluster.routing.allocation.node_concurrent_recoveries: 16
cluster.routing.allocation.node_initial_primaries_recoveries: 16
======node-2======
cluster.name: cluster-es
node.name: node-2
network.host: 0.0.0.0
node.master: true
node.data: true
http.port: 9200
http.cors.allow-origin: "*"
http.cors.enabled: true
transport.port: 9300
http.max_content_length: 200mb
cluster.initial_master_nodes: ["node-1"]
discovery.seed_hosts: ["192.168.72.129","192.168.72.130"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
cluster.routing.allocation.cluster_concurrent_rebalance: 16
cluster.routing.allocation.node_concurrent_recoveries: 16
cluster.routing.allocation.node_initial_primaries_recoveries: 16
but when I curl http://192.168.72.129:9200/_cat/nodes
there is only one node to show, how can I solve it?

how to create elasticsearch cluster with docker compose in different servers?

I have 2 servers, and create elasticsearch nodes in the 2 servers. the content of docker-compose.yml files are like these:
es0:
image: elasticsearch:7.6.0
container_name: es0
environment:
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
- 9300:9300
volumes:
- "/mnt/docker/es0/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml"
- "/mnt/docker/es0/data:/usr/share/elasticsearch/data"
- "/mnt/docker/es0/plugins:/usr/share/elasticsearch/plugins"
- "/mnt/docker/es0/config/cert:/usr/share/elasticsearch/config/cert"
es1:
image: elasticsearch:7.6.0
container_name: es1
environment:
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
- 9300:9300
volumes:
- "/mnt/docker/es1/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml"
- "/mnt/docker/es1/data:/usr/share/elasticsearch/data"
- "/mnt/docker/es1/plugins:/usr/share/elasticsearch/plugins"
- "/mnt/docker/es1/config/cert:/usr/share/elasticsearch/config/cert"
and I configured the elasticsearch.yml like these:
cluster.name: hs-cluster
node.name: es-00
node.master: true
node.data: true
http.host: 0.0.0.0
http.port: 9200
transport.host: 0.0.0.0
transport.tcp.port: 9300
#network.host: 0.0.0.0
network.bind_host: ["192.168.0.2", "101.xx.xx.136"]
network.publish_host: 192.168.0.2
gateway.recover_after_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
cluster.initial_master_nodes: ["es-00", "es-01"]
discovery.seed_hosts: [ "192.168.0.2:9300", "192.168.0.3:9300" ]
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
cluster.name: hs-cluster
node.name: es-01
node.master: true
node.data: true
http.host: 0.0.0.0
http.port: 9200
transport.host: 0.0.0.0
transport.tcp.port: 9300
#network.host: 0.0.0.0
network.bind_host: ["192.168.0.3", "101.xx.xx.137"]
network.publish_host: 192.168.0.3
gateway.recover_after_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
cluster.initial_master_nodes: ["es-00", "es-01"]
discovery.seed_hosts: [ "192.168.0.2:9300", "192.168.0.3:9300" ]
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
when I run the instances, they all started successfully. But when I call _cluster/state?pretty, they all gave the error message:
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
that means they can't find each other.
I also tried to set network.host: 0.0.0.0
but the result was the same.
Who know the reason of this master not discovered exception? How to resolve it?
btw, I can ran the cluster in the same server with docker compose. But in different servers, it is failed. I also ran telnet xxx 9300 in each server, they all connected.
What is your default docker-engine network configuration?
Sometimes multiple servers have the same network, so dockers don't route from one server to another.
To resolve this you have to modify the daemon.json file to the following:
node1
{
"bip": "10.40.18.1/28"
}
node2
{
"bip": "10.40.18.65/28"
}

How to setup Three Machines with different IP Address?

I have three machines, installed same ELK(6.2.2) version in all machine,
One is master and another two's are client node,
** Each machine is the different IP address
I have tried like this, but not working
Server:
cluster.name: sever
node.name: main-server
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
path.data: E:/ELK-6.2.2/elasticsearch/data
path.logs: E:/ELK-6.2.2/elasticsearch/logs
network.host: 11.xx.xx.xx
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["11.XX.XX.XX", "12.xx.xx.xx:9200", "13.xx.xx.xx:9200"]
discovery.zen.minimum_master_nodes: 1
Client:1
cluster.name: client-one
node.name: client-node-one
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 1
path.data: E:/ELK-6.2.2/elasticsearch/data
path.logs: E:/ELK-6.2.2/elasticsearch/logs
network.host: 12.xx.xx.xx
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["11.XX.XX.XX", "12.xx.xx.xx:9200", "13.xx.xx.xx:9200"]
discovery.zen.minimum_master_nodes: 1
Client: 2
cluster.name: client-two
node.name: client-node-two
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 1
path.data: E:/ELK-6.2.2/elasticsearch/data
path.logs: E:/ELK-6.2.2/elasticsearch/logs
network.host: 13.xx.xx.xx
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["11.XX.XX.XX", "12.xx.xx.xx:9200", "13.xx.xx.xx:9200"]
discovery.zen.minimum_master_nodes: 1
Please guide me how to setup these machines?
cluster.name must be equal for all your hosts.

elasticsearch in docker cluster name is same but not clustering

$ curl localhost:9200/_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1469016273 12:04:33 jh-cluster green 1 1 0 0 0 0 0 0 - 100.0%
$ curl localhost:9201/_cat/health?v
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1469016277 12:04:37 jh-cluster green 1 1 0 0 0 0 0 0 - 100.0%
I don't know why this situation. Nodes's config file is same,
paste your elastic configuration, btw this is my configuration for cluster in local (1 master-for-write 1 slave-for-read):
master node:
cluster.name: musement
network.bind_host: 0.0.0.0
transport.tcp.port: 9301
http.port: 9201
node.master: true
node.data: false
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["pippo.int:9300"]
slave node:
cluster.name: musement
network.bind_host: 0.0.0.0
transport.tcp.port: 9300
http.port: 9200
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["pippo.int:9301"]
where pippo.int it's the eth0 of my pc

Can't start elasticsearch an node slave

I cant't start elasticsearch with node.master:false
elasticsearch.yml
cluster.name: graylog2
node.name: "second"
node.master: false
node.data: true
index.number_of_shards: 2
bootstrap.mlockall: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: 192.168.93.76
script.disable_dynamic: true
service elasticsearch restart
netstat -an | grep 9200
NULL
YML has very strict syntax, you need to add a space between node.master and false:
node.master: false

Resources