Can't change vm.max_map_count on LXC - elasticsearch

I try running Elasticsearch 6.x on LXC using Ansible, when try to start Elasticsearch service in elasticsearch log I see:
[2020-01-04T08:45:58,744][ERROR][o.e.b.Bootstrap ] [4WUODd8] node validation exception
[1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
when I searched for this I found :
sysctl -w vm.max_map_count=262144
but When I running this I get :
sysctl: setting key "vm.max_map_count": Read-only file system
I tried change file manually but didn't work , Is there any way change this environment variable in LXC?

Solved by executing command :
sysctl -w vm.max_map_count=262144
on Ansible hosts it self

Related

Increase vm.max_map_count permanently in podman machine vm

I am trying to run the Elasticsearch image on my Mac m1 laptop using podman. It will fail with the message:
ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
I can solve this temporarily by using ssh to enter the podman machine vm and setting vm.max_map_count with
sudo sysctl -w vm.max_map_count=262144
However, if I stop and restart the podman machine vm, the setting is back to the default 65530.
Is there a way to permanently set vm.max_map_count for the podman machine vm?
I came across this recently too. You make have already sorted it out.
To modify kernel parameters at boot just add an entry do one of your sysctl configs and it will be applied on reboot.
# echo "vm.max_map_count = 262144" >> /etc/sysctl.d/99-sysctl.conf

vm.max_map_count is too low

I am moving from elasticsearch 2.x to 5.x and facing this problem while starting up.
[2017-02-28T14:38:24,490][INFO ][o.e.b.BootstrapChecks ] [node1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-02-28T14:38:24,494][ERROR][o.e.b.Bootstrap ] [node1] node validation exception
bootstrap checks failed
max file descriptors [8192] for elasticsearch process is too low, increase to at least [65536]
max size virtual memory [52729364480] for user [elastic] is too low, increase to [unlimited]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
My yml looks as follows
node.name: node1
network.host: _global_
I have downloaded the elasticsearch tarball and OS is SLES11Sp4
Elasticsearch bootstrap checks are defined here. But after playing around with /etc/security/limits.conf and /etc/sysctl.conf, still i cant bring it up.
In rpm installation, these things are supposedly taken care automatically.
This is the setting which finally worked.
/etc/security/limits.conf
* hard memlock unlimited
* soft memlock unlimited
* hard nofile 65536
* soft nofile 65536
* - as unlimited
/etc/sysctl.conf
fs.file-max = 2097152
vm.max_map_count = 262144
vm.swappiness = 1
elasticsearch.yml
cluster.name: atul-es-kerberos
node.name: node1
network.host: _eth0:ipv4_
discovery.zen.ping.unicast.hosts: ["atul.labs.com"]
discovery.zen.minimum_master_nodes: 1
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
1. Linux
The vm.max_map_count=262144 setting should be set permanently in /etc/sysctl.conf
grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144
OR
sysctl -w vm.max_map_count=262144
For Docker on Windows, you should:
docker-machine ssh
sudo sysctl -w vm.max_map_count=262144

Running Elastic Search with Laradock

I'm trying to get ElasticSearch running with Laradock. ES looks to be supported out of the box with Laradock.
Here's my docker command (run from <project root>/laradock/:
docker-compose up -d nginx postgres redis beanstalkd elasticsearch
However if I run docker ps, the elasticsearch container isn't running.
Both ports 9200 and 9300 are not consumed:
lsof -i :9200
Not sure why the elasticsearch container doesn't persist, it seems to just self close.
output of docker ps -a after running docker-compose up ...
http://pastebin.com/raw/ymfvLPLT
Condensed version:
IMAGE STATUS PORTS
laradock_nginx Up 36 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp
laradock_elasticsearch Exited (137) 34 seconds ago
laradock_beanstalkd Up 37 seconds 0.0.0.0:11300->11300/tcp
laradock_php-fpm Up 38 seconds 9000/tcp
laradock_workspace Up 39 seconds 0.0.0.0:2222->22/tcp
tianon/true Excited (0) 41 seconds ago
laradock_postgres Up 41 seconds 0.0.0.0:5432->5432/tcp
laradock_redis Up 40 seconds 0.0.0.0:6379->6379/tcp
Output of docker events after running docker-compose up ...
http://pastebin.com/cE9bjs6i
Try to check logs first:
docker logs laradock_elasticsearch_1
(or another name of elasticsearch container)
In my case it was
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
I found solution here
namely, i've run on my Ubuntu machine
sudo sysctl -w vm.max_map_count=262144
I don't think the problem is related to Laradock, since Elasticsearch is supposed to be running on it's own, I would first check the memory:
open Docker Dashboard -> Settings -> Resources -> Advanced: and increase the memory.
check your Machine memory, Elasticsearch won't run if there is not enough memory in your machine.
or:
open your docker-compose.yml file
increase the mem_limit: 1g then
docker-compose up -d --build elasticsearch
If it's still not working, remove all the images, update laradock to latest version and setup it new.

Docker - ELK - vm.max_map_count

I'm trying to use the docker's image elk-docker (https://elk-docker.readthedocs.io/) , using Docker Compose. The .yml file, is like this:
elk:
image: sebp/elk
ports:
- "5601:5601"
- "9200:9200"
- "5044:5044"
When I run the command: sudo docker-compose up, the console shows:
* Starting Elasticsearch Server
sysctl: setting key "vm.max_map_count": Read-only file system
...fail!
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.
cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory
docker_elk_1 exited with code 1
How can I resolve the problem?
You can do that in 2 ways.
Temporary set max_map_count:
sudo sysctl -w vm.max_map_count=262144
but this will only last till you restart your system.
Permament
In your host machine
vi /etc/sysctl.conf
make entry
vm.max_map_count=262144
restart
You probably need to set vm.max_map_count in /etc/sysctl.conf on the host itself, so that Elasticsearch does not attempt to do that from inside the container. If you don't know the desired value, try doubling the current setting and keep going until Elasticsearch starts successfully. Documentation recommends at least 262144.
In Docker host terminal (Docker CLI) run commands:
docker-machine ssh
sudo sysctl -w vm.max_map_count=262144
exit
Go inside your docker container
# docker exec -it <container_id> /bin/bash
You can view your current max_map_count value
# more /proc/sys/vm/max_map_count
Temporary set max_map_count value(container(host) restart will not persist max_map_count value)
# sysctl -w vm.max_map_count=262144
Permanently set value
1. # vi /etc/sysctl.conf
vm.max_map_count=262144
2. # vi /etc/rc.local
#put parameter inside rc.local file
echo 262144 > /proc/sys/vm/max_map_count
You have to set the vm.max_map_count variable in /etc/sysctl.conf at least with 262144
After that you can reload the settings with sysctl --system

Elasticsearch: Failed to connect to localhost port 9200 - Connection refused

When I tried connecting to Elasticsearch using the
curl http://localhost:9200 it is working fine.
But when I run the curl http://IpAddress:9200 it is throwing an error saying
Failed to connect to localhost port 9200: Connection refused
How to resolve this error?
Edit /etc/elasticsearch/elasticsearch.yml and add the following line:
network.host: 0.0.0.0
This will "unset" this parameter and will allow connections from other IPs.
By default it should bind to all local addresses. So, assuming you don't have a network layer issue with firewalls, the only ES setting I can think to check is network.bind_host and make sure it is either not set or is set to 0.0.0.0 or ::0 or to the correct IP address for your network.
Update: per comments in ES 2.3 you should set network.host instead.
In my case elasticsearch was started.
But still had
curl: (7) Failed to connect to localhost port 9200: Connection refused
The following command was unsuccessful
sudo service elasticsearch restart
In order to make it work, I had to run instead
sudo systemctl restart elasticsearch
Then it went all fine.
Tried everything on this page, and only instructions from here helped.
in /etc/default/elasticsearch, make sure these are un-commented:
START_DAEMON=true
ES_USER=elasticsearch
ES_GROUP=elasticsearch
LOG_DIR=/var/log/elasticsearch
DATA_DIR=/var/lib/elasticsearch
WORK_DIR=/tmp/elasticsearch
CONF_DIR=/etc/elasticsearch
CONF_FILE=/etc/elasticsearch/elasticsearch.yml
RESTART_ON_UPGRADE=true
make sure /var/lib/elasticsearch is owned by elasticsearch user:
chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/
Why don't you start with this command-line:
$ sudo service elasticsearch status
I did it and get:
"There is insufficient memory for the Java Runtime..."
Then I edited /etc/elasticsearch/jvm.options file:
...
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
#-Xms2g
#-Xms2g
-Xms512m
-Xmx512m
################################################################
...
This worked like a charm.
None of the proposed solutions here worked for me, but what eventually got it working was adding the following to elasticsearch.yml
network:
host: 0.0.0.0
http:
port: 9200
After that, I restarted the service and now I can curl it from both within the VM and externally. For some odd reason, I had to try a few different variants of a curl call inside the VM before it worked:
curl localhost:9200
curl http://localhost:9200
curl 127.0.0.1:9200
Note: I'm using Elasticsearch 5.5 on Ubuntu 14.04
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)
be sure that the server is started. I've seen this problem when my virtual machine had too litle RAM and es could not start.
sudo systemctl status elasticsearch
the above will show you if es is indeed running.
Edit elasticsearch.yml and add the following line
http.host: 0.0.0.0
network.host: 0.0.0.0 didn't work for
For this problem, I had to use :
sudo /usr/share/elasticsearch/bin/elasticsearch start
to be able to get something on ports 9200/9300 (sudo netstat -ntlp) and a response to:
curl -XGET http://localhost:9200
I experienced a similar issue.
Here's how I solved it
Run the service command below to start ElasticSearch
sudo service elasticsearch start
OR
sudo systemctl start elasticsearch
If you still get the error
curl: (7) Failed to connect to localhost port 9200: Connection refused
Run the service command below to check the status of ElasticSearch
sudo service elasticsearch status
OR
sudo systemctl status elasticsearch
If you get a response (Active: active (running)) like the one below then you ElasticSearch is active and running
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2019-09-21 11:22:21 WAT; 3s ago
You can then test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost using the command below:
curl http://localhost:9200
Else, if you get a response a different response, you may have to debug further to fix it, but the running the command below, will help you detect what caveats are holding ElasticSearch service from starting.
sudo service elasticsearch status
OR
sudo systemctl status elasticsearch
If you want to stop the ElasticSearch service, simply run the service command below;
sudo service elasticsearch stop
OR
sudo systemctl stop elasticsearch
N/B: You may have to run the command sudo service elasticsearch status OR sudo systemctl status elasticsearch each time you encounter the error, in order to tell the state of the ElasticSearch service.
This also applies for Kibana, run the command sudo service kibana status OR sudo systemctl status kibana each time you encounter the error, in order to tell the state of the Kibana service.
That's all.
I hope this helps.
I had the same problem refusing connections on 9200 port.
Check elasticsearch service status with the command sudo service elasticsearch status. If it is presenting an error and you read anything related to Java, probably the problem is your jvm memory. You can edit it in /etc/elasticsearch/jvm.options. For a 1GB RAM memory machine on Amazon environment, I kept my configuration on:
-Xms128m
-Xmx128m
After setting that and restarting elasticsearch service, it worked like a charm. Nmap and UFW (if you use local firewall) checking should also be useful.
Open your Dockerfile under elasticsearch folder and update "network.host=0.0.0.0" with "network.host=127.0.0.1". Then restart the container. Check your connection with curl.
$ curl http://docker-machine-ip:9200
{
"name" : "vI6Zq_D",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "hhyB_Wa4QwSX6zZd1F894Q",
"version" : {
"number" : "5.2.0",
"build_hash" : "24e05b9",
"build_date" : "2017-01-24T19:52:35.800Z",
"build_snapshot" : false,
"lucene_version" : "6.4.0"
},
"tagline" : "You Know, for Search"
}
For versions higher than 6.8 (7.x) you need two things.
1. change the network host to listen on the public interface.
In the configuration file elasticsearch.yml (for debian and derivatives -> /etc/elasticsearch/elasticsearch.yml).
set the network.host or network.bind_host to:
...
network.host: 0.0.0.0
...
Or the interface that must be reached
2. Before going to production it's necessary to set important discovery and cluster formation settings.
According to elastic.co:
v6.8 -> discovery settings that should set.
by e.g
...
# roughly means the same as 1
discovery.zen.minimum_master_nodes: -1
...
v7.x -> discovery settings that should set.
by one single node
discovery.type: single-node
#OR set discovery.seed_hosts : 127.0.0.1:9200
at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured.
In this case, first of all you need to check the java version using below command:
java -version
after running this command you get something like this:
java version "1.7.0_51"
OpenJDK Runtime Environment (rhel-2.4.5.5.el7-x86_64 u51-b31)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
then use this command:
update-alternatives --config java
and select the below version
*+ 1 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64/jre/bin/java
2 /usr/java/jdk1.8.0_73/jre/bin/java
Enter to keep the current selection[+], or type selection number: 2
curl -XGET http://127.0.0.1:9200
My 2 cents,
I just followed the install procedure on Digital Ocean, apparently the package available in the repos is not up to date, I deleted everything and followed the install procedure direct from Elastic Search and everything is working now, basically the out of the box behaviour is on a localhost pointing to 9200. Same thing/issue found with Kibana, the solution for me was too, to remove everything and just follow their procedure, Hope this saves someone two hours (the time I spent figuring out how to setup ELK!)
en
Update your jdk to latest minimum version for your elasticsearch.
Change the network.bind to 0.0.0.0 and http:port to 9200. The bind address 0.0.0.0 means all IPv4 addresses on the local machine. If a host has two IP addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs.
If you encounter the Connection refused error, simply run the command below to check the status of ElasticSearch service
sudo service elasticsearch status
This will help you decipher the state of ElasticSearch service and what to do about it.
For those of you installing ELK on virtual machine in GCP (Google Cloud Platform), make sure that you created firewall rule of Ingress type (i.e. for incoming to VM traffic). You can specify in the rule multiple ports at a time by separating them with comma: 5000,5044,5601,9200,9300,9600.
In that rule you may want to specify a tag (pick tag's name as you like, for example docker-elk that will target your VM (Targets column):
On VM's settings page assign that tag to your VM:
After doing that I was able to access Elasticsearch in my browser via port 9200. And I didn't have to edit elasticsearch.yml file whatsoever.
I have run across this problem every time I install or upgrade ES (7.0+). And the solution was ALWAYS just wait for ES to fully start. It takes about a minute for the REST API to be reponsive. No matter what service status says.
service elasticsearch start
*started
*wait for at least a minute
curl now works and returns responses on the port 9200
After utilizing some of the answers above, don't forget that after an apt install, a total reboot might be in order.
Just to add on this, I've came across many docs through google that said to set network.host to localhost.
Doing so gave me the infamous connection refused. You must use an IP address (127.0.0.1), not a FQDN.
Jeff
Make sure that port 9200 is open for my case it was an amazon instance so when i opened it in my security group the curl command worked.
Disabling SELinux worked for me, although I don't suggest it - I did that just for a PoC
My problem was I could not work with localhost I needed to set it to localhost's IP address
network.bind_host: 127.0.0.1
In my case, the problem is with java version, i installed open-jdk 11 previously. Thats creating the issue while starting the service. I changed it open-jdk 8 and it started working
I experienced this on CentOS 7, and the issue was that /etc/hosts had the following:
127.0.0.1 localhost.localdomain
which I updated to include localhost as follows:
127.0.0.1 localhost localhost.localdomain
after that, no issues.
you have to edit /etc/elasticsearch/elasticsearch.yml
by default all configurations will be commented ,add following configuration
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [0.0.0.0]
then restart the service
I ran into a related situation recently.
Here's my take on the subject: Accessing Elastic 5.5 in vagrant guest from host through a private network
TL;DR
The settings:
network.host: 0.0.0.0
http.port: 9200
work fine. One just needs to wait enough time for ES to complete its initialization procedure, bind to the network iface and start listening on the port.
Now, from within the guest, curl http://localhost:9200 works and from the host, curl http://192.168.54.2:9200 works as well.
For Windows user try,
https://localhost:9200/
It worked for me.

Resources