vm.max_map_count is too low - elasticsearch

I am moving from elasticsearch 2.x to 5.x and facing this problem while starting up.
[2017-02-28T14:38:24,490][INFO ][o.e.b.BootstrapChecks ] [node1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-02-28T14:38:24,494][ERROR][o.e.b.Bootstrap ] [node1] node validation exception
bootstrap checks failed
max file descriptors [8192] for elasticsearch process is too low, increase to at least [65536]
max size virtual memory [52729364480] for user [elastic] is too low, increase to [unlimited]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
My yml looks as follows
node.name: node1
network.host: _global_
I have downloaded the elasticsearch tarball and OS is SLES11Sp4
Elasticsearch bootstrap checks are defined here. But after playing around with /etc/security/limits.conf and /etc/sysctl.conf, still i cant bring it up.
In rpm installation, these things are supposedly taken care automatically.

This is the setting which finally worked.
/etc/security/limits.conf
* hard memlock unlimited
* soft memlock unlimited
* hard nofile 65536
* soft nofile 65536
* - as unlimited
/etc/sysctl.conf
fs.file-max = 2097152
vm.max_map_count = 262144
vm.swappiness = 1
elasticsearch.yml
cluster.name: atul-es-kerberos
node.name: node1
network.host: _eth0:ipv4_
discovery.zen.ping.unicast.hosts: ["atul.labs.com"]
discovery.zen.minimum_master_nodes: 1

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
1. Linux
The vm.max_map_count=262144 setting should be set permanently in /etc/sysctl.conf
grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144
OR
sysctl -w vm.max_map_count=262144

For Docker on Windows, you should:
docker-machine ssh
sudo sysctl -w vm.max_map_count=262144

Related

Unable to start Elasticsearch after fresh install

I have been trying to install elasticsearch without success. Every single guide regardless of if I used the .deb or install from apt fails in the same way. I am using ubuntu 20.04 as an LXC on proxmox. After installing the service start fails and here is the log file
[2022-06-10T12:06:13.884+0000][22320][gc,init] CardTable entry size: 512
[2022-06-10T12:06:13.885+0000][22320][gc ] Using G1
[2022-06-10T12:06:13.957+0000][22320][gc,init] Version: 18.0.1.1+2-6 (release)
[2022-06-10T12:06:13.957+0000][22320][gc,init] CPUs: 24 total, 4 available
[2022-06-10T12:06:13.957+0000][22320][gc,init] Memory: 96593M
[2022-06-10T12:06:13.957+0000][22320][gc,init] Large Page Support: Disabled
[2022-06-10T12:06:13.957+0000][22320][gc,init] NUMA Support: Disabled
[2022-06-10T12:06:13.958+0000][22320][gc,init] Compressed Oops: Enabled (Non-zero disjoint base)
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Region Size: 16M
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Min Capacity: 31G
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Initial Capacity: 31G
[2022-06-10T12:06:13.958+0000][22320][gc,init] Heap Max Capacity: 31G
[2022-06-10T12:06:13.958+0000][22320][gc,init] Pre-touch: Disabled
[2022-06-10T12:06:13.958+0000][22320][gc,init] Parallel Workers: 4
[2022-06-10T12:06:13.958+0000][22320][gc,init] Concurrent Workers: 1
[2022-06-10T12:06:13.958+0000][22320][gc,init] Concurrent Refinement Workers: 4
[2022-06-10T12:06:13.958+0000][22320][gc,init] Periodic GC: Disabled
[2022-06-10T12:06:13.958+0000][22320][gc,metaspace] CDS archive(s) not mapped
[2022-06-10T12:06:13.958+0000][22320][gc,metaspace] Compressed class space mapped at: 0x0000000080000000-0x00000000c0000000, reserved size: 1073741>
[2022-06-10T12:06:13.958+0000][22320][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 0, Narrow klass range: 0xc0000000
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] Heap
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] garbage-first heap total 32505856K, used 24578K [0x0000001001000000, 0x00000017c1000000)
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] region size 16384K, 2 young (32768K), 0 survivors (0K)
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] Metaspace used 3525K, committed 3584K, reserved 1114112K
[2022-06-10T12:06:14.152+0000][22320][gc,heap,exit] class space used 271K, committed 320K, reserved 1048576K
So pretty much all online guides to install elastic search are useless except this one:
https://techviewleo.com/install-elastic-stack-elk-8-on-ubuntu/
For anyone attempting to install elastic search and comes here the main thing this above guide does that the other ones dont is it disables all the security in the yaml file:
Here are the settings that worked for me
network.host: localhost
cluster.name: my-application
node.name: node-1
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
As warning this config should not be used for a publicly exposed instance

Increase vm.max_map_count permanently in podman machine vm

I am trying to run the Elasticsearch image on my Mac m1 laptop using podman. It will fail with the message:
ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
I can solve this temporarily by using ssh to enter the podman machine vm and setting vm.max_map_count with
sudo sysctl -w vm.max_map_count=262144
However, if I stop and restart the podman machine vm, the setting is back to the default 65530.
Is there a way to permanently set vm.max_map_count for the podman machine vm?
I came across this recently too. You make have already sorted it out.
To modify kernel parameters at boot just add an entry do one of your sysctl configs and it will be applied on reboot.
# echo "vm.max_map_count = 262144" >> /etc/sysctl.d/99-sysctl.conf

Can't change vm.max_map_count on LXC

I try running Elasticsearch 6.x on LXC using Ansible, when try to start Elasticsearch service in elasticsearch log I see:
[2020-01-04T08:45:58,744][ERROR][o.e.b.Bootstrap ] [4WUODd8] node validation exception
[1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
when I searched for this I found :
sysctl -w vm.max_map_count=262144
but When I running this I get :
sysctl: setting key "vm.max_map_count": Read-only file system
I tried change file manually but didn't work , Is there any way change this environment variable in LXC?
Solved by executing command :
sysctl -w vm.max_map_count=262144
on Ansible hosts it self

master_not_discovered_exception ElasticSearch single node

I am trying to setup elasticsearch on a single host. Here is how my configuration looks like:
elasticsearch.yml
node.name: ${HOSTNAME}
network.host: _site_, _local_
http.port: 9200
transport.tcp.port: 9300
cluster.name: "test_cluster"
node.local: true
kibana.yml
server.host: 0.0.0.0
elasticsearch.url: http://localhost:9200
On following command:
curl -XGET 'localhost:9200/_cluster/health?pretty'
I get following message:
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
In log file I see following message:
not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
Could someone please point me right direction here?
I spent days (sigh) on basically this. I was trying to upgrade my single node cluster from 6.x es to 7.x, and I kept dying on the hill of "master_not_discovered_exception".
What finally solved it for me was examining a completely fresh install of 7.x.
For my single-node cluster, my /etc/elasticsearch/elasticsearch.yml needed the line:
discovery.type: single-node
I hope that saves someone else from spending days like I did. In my defence, I'm very new to es.
Following changes in eleasticsearch.yml worked for me
Node
node.name: node-1
Paths
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
Network
network.host: 0.0.0.0
http.port: 9200
Discovery
discovery.seed_hosts: "127.0.0.1:9300"
cluster.initial_master_nodes: ["node-1"]
Steps:
sudo systemctl stop elasticsearch.service
sudo nano /etc/elasticsearch/elasticsearch.yml
Apply above changes, save and close the editor.
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
NOTE: elasticsearch7 is used with ubuntu 20.04, it should work with other versions as well
According to this link, for the very first time you should set initial_master_nodes config like this in /etc/elasticsearch/elasticsearch.yml:
node.name: "salehnode"
cluster.initial_master_nodes:
- salehnode
On my side, after installing ES 7+, I had to set node-1 as a master-eligible node in the new cluster.
sudo nano /etc/elasticsearch/elasticsearch.yml
cluster.initial_master_nodes: ["node-1"]
First I dont think you need network.host setting for this.
In your log it is trying to get a master, but result is 0
Can you try setting properties like:
node.master: true
node.data:true
Also can you please put more logs here, if it doesnt work
You can try this in your elasticsearch.yml:
discovery.zen.minimum_master_nodes: 1
I think ES will always find discovery.zen.ping.unicast.hosts until it
finds nodes of count equal to minimum_master_nodes.

Docker - ELK - vm.max_map_count

I'm trying to use the docker's image elk-docker (https://elk-docker.readthedocs.io/) , using Docker Compose. The .yml file, is like this:
elk:
image: sebp/elk
ports:
- "5601:5601"
- "9200:9200"
- "5044:5044"
When I run the command: sudo docker-compose up, the console shows:
* Starting Elasticsearch Server
sysctl: setting key "vm.max_map_count": Read-only file system
...fail!
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.
cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory
docker_elk_1 exited with code 1
How can I resolve the problem?
You can do that in 2 ways.
Temporary set max_map_count:
sudo sysctl -w vm.max_map_count=262144
but this will only last till you restart your system.
Permament
In your host machine
vi /etc/sysctl.conf
make entry
vm.max_map_count=262144
restart
You probably need to set vm.max_map_count in /etc/sysctl.conf on the host itself, so that Elasticsearch does not attempt to do that from inside the container. If you don't know the desired value, try doubling the current setting and keep going until Elasticsearch starts successfully. Documentation recommends at least 262144.
In Docker host terminal (Docker CLI) run commands:
docker-machine ssh
sudo sysctl -w vm.max_map_count=262144
exit
Go inside your docker container
# docker exec -it <container_id> /bin/bash
You can view your current max_map_count value
# more /proc/sys/vm/max_map_count
Temporary set max_map_count value(container(host) restart will not persist max_map_count value)
# sysctl -w vm.max_map_count=262144
Permanently set value
1. # vi /etc/sysctl.conf
vm.max_map_count=262144
2. # vi /etc/rc.local
#put parameter inside rc.local file
echo 262144 > /proc/sys/vm/max_map_count
You have to set the vm.max_map_count variable in /etc/sysctl.conf at least with 262144
After that you can reload the settings with sysctl --system

Resources