ElasticSearch is swapping - elasticsearch
I have launch my new site today and have problems with the SWAP of ElasticSearch. I don't know how to fix it because I have give enought heap memory for each of the 2 nodes, but somehow, there are problems.
I am attaching the previews from HQ. Can someone help me?
---- config /etc/elasticsearch.yml ---
cluster.name: xxx
node.name: xxx
node.data: true
node.master: true
bootstrap.mlockall: true
index.translog.flush_threshold_ops: 50000
index.merge.policy.merge_factor: 5
index.merge.policy.segments_per_tier: 10
# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 1000
threadpool.bulk.queue_size: 30000
# Index pool
threadpool.index.type: fixed
threadpool.index.size: 1000
threadpool.index.queue_size: 10000
index.cache.query.enable: true
indices.cache.query.size: 15%
index.cache.field.expire: 1h
indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire : 1h
indices.cache.filter.size: 15%
index.store.type: mmapfs
transport.tcp.compress: true;
network.bind_host: xxxx
network.publish_host: xxxx
network.host: xxxx
discovery.zen.ping.unicast.hosts: ["xxxx"]
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 10s
transport.tcp.port: 9300
http.port: 9200
http.max_content_length: 500mb
index.routing.allocation.disable_allocation: false
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms
script.engine.groovy.inline.aggs: on
script.inline: on
script.indexed: on
index.max_result_window: 40000
--- config /etc/default/elasticsearch -----
ES_HEAP_SIZE=5g
---- JAVA process -----
/usr/bin/java -Xms5g -Xmx5g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.1.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
---- node info ----
{"cluster_name":"xxxxx_search","nodes":{"Tze_hXZ2SwqQIvg4YWOcMg":{"timestamp":1460563870399,"name":"xxxxx-slave-1","transport_address":"xxxx:9301","host":"xxxxx","ip":["xxxxx:9301","NONE"],"jvm":{"timestamp":1460563870399,"uptime_in_millis":17908517,"mem":{"heap_used_in_bytes":2679603328,"heap_used_percent":50,"heap_committed_in_bytes":5333843968,"heap_max_in_bytes":5333843968,"non_heap_used_in_bytes":105721416,"non_heap_committed_in_bytes":108711936,"pools":{"young":{"used_in_bytes":15540664,"max_in_bytes":279183360,"peak_used_in_bytes":279183360,"peak_max_in_bytes":279183360},"survivor":{"used_in_bytes":2730400,"max_in_bytes":34865152,"peak_used_in_bytes":34865152,"peak_max_in_bytes":34865152},"old":{"used_in_bytes":2661332264,"max_in_bytes":5019795456,"peak_used_in_bytes":3813217632,"peak_max_in_bytes":5019795456}}},"threads":{"count":56,"peak_count":63},"gc":{"collectors":{"young":{"collection_count":1170,"collection_time_in_millis":38743},"old":{"collection_count":4,"collection_time_in_millis":220}}},"buffer_pools":{"direct":{"count":61,"used_in_bytes":15744627,"total_capacity_in_bytes":15744627},"mapped":{"count":85,"used_in_bytes":4319218096,"total_capacity_in_bytes":4319218096}}}},"DrG535FuQKygzKlSCAWwLw":{"timestamp":1460563870399,"name":"xxxxx-master-1","transport_address":"xxxxx:9300","host":"xxxx","ip":["xxxxx:9300","NONE"],"attributes":{"master":"true"},"jvm":{"timestamp":1460563870399,"uptime_in_millis":17912689,"mem":{"heap_used_in_bytes":2315059272,"heap_used_percent":43,"heap_committed_in_bytes":5333843968,"heap_max_in_bytes":5333843968,"non_heap_used_in_bytes":118353088,"non_heap_committed_in_bytes":121683968,"pools":{"young":{"used_in_bytes":172840784,"max_in_bytes":279183360,"peak_used_in_bytes":279183360,"peak_max_in_bytes":279183360},"survivor":{"used_in_bytes":2480072,"max_in_bytes":34865152,"peak_used_in_bytes":34865152,"peak_max_in_bytes":34865152},"old":{"used_in_bytes":2139738416,"max_in_bytes":5019795456,"peak_used_in_bytes":3826731840,"peak_max_in_bytes":5019795456}}},"threads":{"count":59,"peak_count":71},"gc":{"collectors":{"young":{"collection_count":1368,"collection_time_in_millis":47571},"old":{"collection_count":5,"collection_time_in_millis":270}}},"buffer_pools":{"direct":{"count":71,"used_in_bytes":24539898,"total_capacity_in_bytes":24539898},"mapped":{"count":84,"used_in_bytes":4318926707,"total_capacity_in_bytes":4318926707}},"classes":{"current_loaded_count":9552,"total_loaded_count":9695,"total_unloaded_count":143}}}}}
---- node process ---
{"cluster_name":"xxxx_search","nodes":{"Tze_hXZ2SwqQIvg4YWOcMg":{"name":"xxxx-slave-1","transport_address":"xxxx:9301","host":"xxxx","ip":"xxxx","version":"2.3.1","build":"bd98092","http_address":"xxxx:9201","process":{"refresh_interval_in_millis":1000,"id":25686,"mlockall":false}},"DrG535FuQKygzKlSCAWwLw":{"name":"xxxx-master-1","transport_address":"xxxx:9300","host":"xxxx","ip":"xxxx","version":"2.3.1","build":"bd98092","http_address":"xxxx:9200","attributes":{"master":"true"},"process":{"refresh_interval_in_millis":1000,"id":25587,"mlockall":false}}}}
Thanks!
Related
How do I connect two nodes in elasticsearch?
Now I have two nodes(192.168.72.129, 192.168.72.130) It's the setting in config/elasticsearch.yml ======node-1====== cluster.name: cluster-es node.name: node-1 network.host: 0.0.0.0 node.master: true node.data: true http.port: 9200 http.cors.allow-origin: "*" http.cors.enabled: true transport.port: 9300 http.max_content_length: 200mb cluster.initial_master_nodes: ["node-1"] discovery.seed_hosts: ["192.168.72.129","192.168.72.130"] gateway.recover_after_nodes: 2 network.tcp.keep_alive: true network.tcp.no_delay: true transport.tcp.compress: true cluster.routing.allocation.cluster_concurrent_rebalance: 16 cluster.routing.allocation.node_concurrent_recoveries: 16 cluster.routing.allocation.node_initial_primaries_recoveries: 16 ======node-2====== cluster.name: cluster-es node.name: node-2 network.host: 0.0.0.0 node.master: true node.data: true http.port: 9200 http.cors.allow-origin: "*" http.cors.enabled: true transport.port: 9300 http.max_content_length: 200mb cluster.initial_master_nodes: ["node-1"] discovery.seed_hosts: ["192.168.72.129","192.168.72.130"] gateway.recover_after_nodes: 2 network.tcp.keep_alive: true network.tcp.no_delay: true transport.tcp.compress: true cluster.routing.allocation.cluster_concurrent_rebalance: 16 cluster.routing.allocation.node_concurrent_recoveries: 16 cluster.routing.allocation.node_initial_primaries_recoveries: 16 but when I curl http://192.168.72.129:9200/_cat/nodes there is only one node to show, how can I solve it?
how to create elasticsearch cluster with docker compose in different servers?
I have 2 servers, and create elasticsearch nodes in the 2 servers. the content of docker-compose.yml files are like these: es0: image: elasticsearch:7.6.0 container_name: es0 environment: - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" ulimits: memlock: soft: -1 hard: -1 ports: - 9200:9200 - 9300:9300 volumes: - "/mnt/docker/es0/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml" - "/mnt/docker/es0/data:/usr/share/elasticsearch/data" - "/mnt/docker/es0/plugins:/usr/share/elasticsearch/plugins" - "/mnt/docker/es0/config/cert:/usr/share/elasticsearch/config/cert" es1: image: elasticsearch:7.6.0 container_name: es1 environment: - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" ulimits: memlock: soft: -1 hard: -1 ports: - 9200:9200 - 9300:9300 volumes: - "/mnt/docker/es1/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml" - "/mnt/docker/es1/data:/usr/share/elasticsearch/data" - "/mnt/docker/es1/plugins:/usr/share/elasticsearch/plugins" - "/mnt/docker/es1/config/cert:/usr/share/elasticsearch/config/cert" and I configured the elasticsearch.yml like these: cluster.name: hs-cluster node.name: es-00 node.master: true node.data: true http.host: 0.0.0.0 http.port: 9200 transport.host: 0.0.0.0 transport.tcp.port: 9300 #network.host: 0.0.0.0 network.bind_host: ["192.168.0.2", "101.xx.xx.136"] network.publish_host: 192.168.0.2 gateway.recover_after_nodes: 1 http.cors.enabled: true http.cors.allow-origin: "*" cluster.initial_master_nodes: ["es-00", "es-01"] discovery.seed_hosts: [ "192.168.0.2:9300", "192.168.0.3:9300" ] bootstrap.memory_lock: true bootstrap.system_call_filter: false cluster.name: hs-cluster node.name: es-01 node.master: true node.data: true http.host: 0.0.0.0 http.port: 9200 transport.host: 0.0.0.0 transport.tcp.port: 9300 #network.host: 0.0.0.0 network.bind_host: ["192.168.0.3", "101.xx.xx.137"] network.publish_host: 192.168.0.3 gateway.recover_after_nodes: 1 http.cors.enabled: true http.cors.allow-origin: "*" cluster.initial_master_nodes: ["es-00", "es-01"] discovery.seed_hosts: [ "192.168.0.2:9300", "192.168.0.3:9300" ] bootstrap.memory_lock: true bootstrap.system_call_filter: false when I run the instances, they all started successfully. But when I call _cluster/state?pretty, they all gave the error message: { "error" : { "root_cause" : [ { "type" : "master_not_discovered_exception", "reason" : null } ], "type" : "master_not_discovered_exception", "reason" : null }, "status" : 503 } that means they can't find each other. I also tried to set network.host: 0.0.0.0 but the result was the same. Who know the reason of this master not discovered exception? How to resolve it? btw, I can ran the cluster in the same server with docker compose. But in different servers, it is failed. I also ran telnet xxx 9300 in each server, they all connected.
What is your default docker-engine network configuration? Sometimes multiple servers have the same network, so dockers don't route from one server to another. To resolve this you have to modify the daemon.json file to the following: node1 { "bip": "10.40.18.1/28" } node2 { "bip": "10.40.18.65/28" }
How to setup Three Machines with different IP Address?
I have three machines, installed same ELK(6.2.2) version in all machine, One is master and another two's are client node, ** Each machine is the different IP address I have tried like this, but not working Server: cluster.name: sever node.name: main-server node.data: true node.ingest: true node.master: true node.max_local_storage_nodes: 1 path.data: E:/ELK-6.2.2/elasticsearch/data path.logs: E:/ELK-6.2.2/elasticsearch/logs network.host: 11.xx.xx.xx http.port: 9200 transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["11.XX.XX.XX", "12.xx.xx.xx:9200", "13.xx.xx.xx:9200"] discovery.zen.minimum_master_nodes: 1 Client:1 cluster.name: client-one node.name: client-node-one node.data: true node.ingest: true node.master: false node.max_local_storage_nodes: 1 path.data: E:/ELK-6.2.2/elasticsearch/data path.logs: E:/ELK-6.2.2/elasticsearch/logs network.host: 12.xx.xx.xx http.port: 9200 transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["11.XX.XX.XX", "12.xx.xx.xx:9200", "13.xx.xx.xx:9200"] discovery.zen.minimum_master_nodes: 1 Client: 2 cluster.name: client-two node.name: client-node-two node.data: true node.ingest: true node.master: false node.max_local_storage_nodes: 1 path.data: E:/ELK-6.2.2/elasticsearch/data path.logs: E:/ELK-6.2.2/elasticsearch/logs network.host: 13.xx.xx.xx http.port: 9200 transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["11.XX.XX.XX", "12.xx.xx.xx:9200", "13.xx.xx.xx:9200"] discovery.zen.minimum_master_nodes: 1 Please guide me how to setup these machines?
cluster.name must be equal for all your hosts.
elasticsearch in docker cluster name is same but not clustering
$ curl localhost:9200/_cat/health?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1469016273 12:04:33 jh-cluster green 1 1 0 0 0 0 0 0 - 100.0% $ curl localhost:9201/_cat/health?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1469016277 12:04:37 jh-cluster green 1 1 0 0 0 0 0 0 - 100.0% I don't know why this situation. Nodes's config file is same,
paste your elastic configuration, btw this is my configuration for cluster in local (1 master-for-write 1 slave-for-read): master node: cluster.name: musement network.bind_host: 0.0.0.0 transport.tcp.port: 9301 http.port: 9201 node.master: true node.data: false discovery.zen.minimum_master_nodes: 1 discovery.zen.ping.timeout: 3s discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["pippo.int:9300"] slave node: cluster.name: musement network.bind_host: 0.0.0.0 transport.tcp.port: 9300 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping.timeout: 3s discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["pippo.int:9301"] where pippo.int it's the eth0 of my pc
Can't start elasticsearch an node slave
I cant't start elasticsearch with node.master:false elasticsearch.yml cluster.name: graylog2 node.name: "second" node.master: false node.data: true index.number_of_shards: 2 bootstrap.mlockall: true discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: 192.168.93.76 script.disable_dynamic: true service elasticsearch restart netstat -an | grep 9200 NULL
YML has very strict syntax, you need to add a space between node.master and false: node.master: false