Using Elasticsearch sink connector to insert data to ES 7.2 instance hosted on VM.
Getting this : Elastic search max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Is it possible to ignore bootstrap checks ?
How can I increase the virtual memory for Elasticsearch in the docker container ?
Bootstrap checks inspect a variety of Elasticsearch and system settings. If youre in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If youre in production mode failed bootstrap will cause Elasticsearch to refuse to start.
Elasticsearch work mode is configured implicit. As soon as you configure a network setting like network.host,Elasticsearch assumes that you are moving to production and will upgrade the above warnings to exceptions.
Regarding your specific case you need to increase it on the host machine not the docker by running this command:
sudo sysctl -w vm.max_map_count=262144 . and then restart your docker-containers.
BTW not recommended but if you are running a single node you can skip the bootstrap checks by not binding transport to an external interface or by binding transport to an external interface and setting the discovery type to single-node.
Related
I have an application up and running in set of docker containers (deployed using docker stack and not kubernetes). I want to do performance monitoring for this application. I am confused about whether I should go for beats or Elastic Agent.
This page says:
When Elastic Agent runs inside of a container, it cannot be upgraded through Fleet as it expects that the container itself is upgraded.
This page says:
Standalone mode — All policies are applied to the Elastic Agent manually as a YAML file.
Q1. Does this mean that in standalone mode Elastic Agent is not "at all" managed by Fleet? Or some part of management
This page says:
Standalone Elastic Agents are manually configured and managed locally on the systems where they are installed. They are useful when you are not interested in centrally managing agents in Fleet, either due to your company’s security requirements, or because you prefer to use another configuration management system.
This page says:
To run an Elastic Agent in standalone mode, install the agent on each host you want to monitor and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents.
Q2. Does this mean that for monitoring docker containers (deployed using docker stack and not kubernetes), there is no difference between between beats and Elastic agents in terms of "central" management? Only difference would be that I have to configure different beats separately, which is avoided with Elastic Agent?
Q3. What is preferrable in this case? Beats or Elastic Agent?
Standalone is not managed by fleet, the "it cannot be upgraded through Fleet" refers to upgrading the actual version of the elastic-agent, you can still update the agent polices.
Yes you would need to configure beats via a config file on the container where elastic agent can be setup with some env variables to enroll it in a policy, then that policy is centrally managed and updated via Kibana.
Both are valid but elastic agent allows updating the policy after the container is running via a central location and therefore would be my choice.
I have configured a mixed node Kubernetes cluster. Two worker nodes are Unbuntu Server 18.04.4 and two worker nodes are Windows Server 2019 Standard. I have deployed several Docker containers as deployments/pods to each set of worker nodes (.NET Core apps on Ubuntu and legacy WCF apps on Windows). Everything seems to work as advertised.
I am now at the point where I want to monitor the resources of the pod/containers. I have deployed Prometheus, kube-state-metrics, metrics-server. I have Prometheus scraping the nodes. For container metrics, the kubelet/cAdvisor is returning everything I need from the Ubunutu nodes, such as "container_cpu_usage_seconds_total, container_cpu_cfs_throttled_seconds_total, etc". But the kubelet/cAdvisor for the Windows nodes only give me some basic information:
http://localhost:8001/api/v1/nodes/[WINDOWS_NODE]/proxy/metrics/cadvisor
# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="",kernelVersion="10.0.17763.1012",osVersion="Windows Server 2019 Standard"} 1
# HELP container_scrape_error 1 if there was an error while getting container metrics, 0 otherwise
# TYPE container_scrape_error gauge
container_scrape_error 0
# HELP machine_cpu_cores Number of CPU cores on the machine.
# TYPE machine_cpu_cores gauge
machine_cpu_cores 2
# HELP machine_memory_bytes Amount of memory installed on the machine.
# TYPE machine_memory_bytes gauge
machine_memory_bytes 1.7179398144e+10
So while the cAdvisor on the Ubuntu nodes gives me everything I ever wanted about containers and more, the cAdvisor on the Windows nodes only gives me the above.
I have examined the Powershell scripts that install/configure kubelet on the Windows nodes, but don't see/understand how I might configure a switch or config file if there is a magical setting I am missing that would enable container metrics to be published when kubelet/cAdvisor is scraped. Any suggestions?
There is metrics/resource/v1alpha1 endpoint. But it provides only 4 basic metrics.
Documentation
I think that cAdvisor doesn't support windows nodes properly that you see is just a n emulated interface with limited metrics
Github issue
As suggested by https://crate.io/docs/crate/guide/en/latest/deployment/cloud/aws/ec2-setup.html. I've been trying to setup a CrateDb (3.0.2) cluster. However setting network.host: _site_ prevents the node from starting. Setting everything back to default settings of course starts the instance but this is bound to localhost.
All I'm trying to do is to evaluate and set up CrateDB cluster. Sadly the error received isn't useful as it seems to complain about heap size (on a fresh Windows install with 8GB ram and empty database). Does anyone know how to set up a production instance ?
are you installing it on amazon web services or on windows? the documentation above is for AWS. here's the correct documentation: https://crate.io/docs/crate/guide/en/latest/scaling/multi-node-setup.html
I'm running Elasticsearch on a non-production RHEL6 server. I only have a regular user account with no root access. I'm in a very locked-down corporate environment so getting root will be time-consuming and I need a work-around.
When I start the process I get these errors:
max file descriptors [8192] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [salimfadhley] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Is there a way to make ElasticSearch ignore this error condition and just start up.
I'm fully aware that ignoring errors is normally considered unwise, however on this occasion I just need to get ES up and running so that I can devote my attention to other aspects of this project: Getting the system limits raised will take more time than I currently have available.
#ThomasDecaux is correct that you can technically disable bootstrap. But you can only disable bootstrap checks sometimes. Here is when you can disable bootstrap checks:
configured your elasticsearch.yml with discovery.type: single-node
or your N nodes are all using zen discovery on localhost
If you need to run N nodes on N machines as one cluster, then no you can not disable bootstrap checks.
When you configure your elasticsearch.yml to use an external interface and you don't have discovery.type: single-node then bootstrap checks can not be disabled. I tried.
I had a machine failing the bootstrap tests but I didn't have sudo permission to fix it. I tried to disable checks by passing the -Des.enforce.bootstrap.checks=false to the Java JVM options but the bootstrap checks were still enabled.
Here is a github issue from 2018 where the developers say you can't disable bootstrap checks https://github.com/elastic/elasticsearch/issues/31933 :
" There is no command line option for disabling bootstrap checks. The
es.enforce.bootstrap.checks option is used to enable them when they
are disabled due to Elasticsearch not detecting that it is being used
in production (single node, only reachable through localhost or using
single-node discovery). "
Yes you can!
(I found) this is very dirty, but if you configure discovery.type as single-node, no bootstrap check will run.
Yes, that means you cannot test a cluster in your laptop.
See https://github.com/elastic/elasticsearch/issues/21655
I have two different machines running elastic search server instances. They automatically create a cluster and changes made on one instance reflect on other instance on different machine. I have changed the cluster.name property in elasticsearch.yml file in config folder and the issue is resolved. I wanted to know if i can start elastic search server instance in non-cluster mode ?
You can't start the es server in non-cluster mode.
But if you want the two servers to run independently (in its own cluster), there are 2 options that I can think of:
Disable multicast and don't set the hosts for them in unicast
Change the cluster.name to make them have different names
The easiest is to set node.local: true
This prevents elasticsearch from trying to connect to other nodes.
Using a custom name is also a good idea in any case just to prevent unintended exchange of data. Use something else for production, testing, and development.