I have a single node elk installation which was working fine up until I discovered that kibana was unable to connect to elasticsearch.
systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2018-12-28 14:21:33 EET; 5min ago
Docs: http://www.elastic.co
Main PID: 1193 (java)
Tasks: 87
Memory: 3.0G
CPU: 5min 39.675s
CGroup: /system.slice/elasticsearch.service
├─1193 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=
└─1548 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
but here is a gist with the exceptions;
for some weird reason after a restart the node seems to recover.
How can this behavior be explained?
Related
I'm running SonarQube on CentOS 7. It is correctly running on the terminal, but if I try to access it through the browser (http:/localhost:9000 or http:/localhost:9001 ) it can't be reached. Can someone help me?
[root#192 logs]# systemctl status sonarqube
● sonarqube.service - SonarQube service
Loaded: loaded (/etc/systemd/system/sonarqube.service; enabled; vendor preset: disabled)
Active: active (running) since ven 2022-10-07 12:48:37 CEST; 3s ago
Process: 23246 ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop (code=exited, status=0/SUCCESS)
Process: 23275 ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start (code=exited, status=0/SUCCESS)
Main PID: 23298 (java)
Tasks: 43
CGroup: /system.slice/sonarqube.service
├─23298 java -Xms8m -Xmx32m --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens...
└─23321 /usr/lib/jvm/java-11-openjdk-11.0.16.1.1-1.el7_9.x86_64/bin/java -XX:+UseG1GC -Djava.io.tmpdir=/opt/sonarqube/temp -XX:ErrorFile...
ott 07 12:48:37 192.168.1.24 systemd[1]: Starting SonarQube service...
ott 07 12:48:37 192.168.1.24 sonar.sh[23275]: /usr/bin/java
ott 07 12:48:37 192.168.1.24 sonar.sh[23275]: Starting SonarQube...
ott 07 12:48:37 192.168.1.24 systemd[1]: Started SonarQube service.
Unable to connect localhost
I am changing the path of -
path.data: /var/log/elasticsearch to path.data: /data/elasticsearchdata/log/elasticsearch/
in elasticsearch.yml
file after creating the folder and moving the files/folders from ../elasticsearch to /data/elasticsearchdata/log/
but after doing the changes in - elasticsearch.yml I have run the command as -
sudo systemctl restart elasticsearch
But getting this error -
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-12-15 14:53:14 UTC; 7s ago
Docs: https://www.elastic.co
Process: 1678664 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 1678664 (code=exited, status=1/FAILURE)
Dec 15 14:53:14 ip-10-10-6-161 systemd-entrypoint[1678664]: path.logs: /data/elasticsearchda ...
Can anyone let me know what I am missing ?
ONLY WAY to move your data is
setup repository (snapshot/restore)
create snapshot of all indices
shut down ELK cluster and edit path.data in elasticsearch.yml
start ELK cluster
restore snapshot
data should appear in the new location
I am installing a brand new elasticsearch 7.5 on OS:Red Hat Enterprise Linux Server release 7.8 (Maipo)
At startup of the service, I have hard failure. here is what the service info provides
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: signal) since Tue 2020-08-25 11:34:39 CEST; 7min ago
Docs: http://www.elastic.co
Process: 102777 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=ABRT)
Main PID: 102777 (code=killed, signal=ABRT)
CGroup: /system.slice/elasticsearch.service
Aug 25 11:34:34 sv-1348lvd44.esante.local systemd[1]: Starting Elasticsearch...
Aug 25 11:34:35 sv-1348lvd44.esante.local elasticsearch[102777]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated...lease.
Aug 25 11:34:39 sv-1348lvd44.esante.local systemd[1]: elasticsearch.service: main process exited, code=killed, status=6/ABRT
Aug 25 11:34:39 sv-1348lvd44.esante.local systemd[1]: Failed to start Elasticsearch.
Aug 25 11:34:39 sv-1348lvd44.esante.local systemd[1]: Unit elasticsearch.service entered failed state.
Aug 25 11:34:39 sv-1348lvd44.esante.local systemd[1]: elasticsearch.service failed.
when using journalctl -xe
Aug 25 11:34:38 sv-1348lvd44.esante.local audispd[824]: node=sv-1348lvd44.esante.local type=ANOM_ABEND msg=audit(1598348078.836:208066): auid=429496 uid=995 gid=991 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=102777 comm="java" reason="memory violation" sig=6
Aug 25 11:34:39 sv-1348lvd44.esante.local systemd[1]: elasticsearch.service: main process exited, code=killed, status=6/ABRT
Aug 25 11:34:39 sv-1348lvd44.esante.local systemd[1]: Failed to start Elasticsearch.
when looking into the dump hs_err_pidXXXX I have.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f4818939b85, pid=52870, tid=52933
#
# JRE version: OpenJDK Runtime Environment (13.0.1+9) (build 13.0.1+9)
# Java VM: OpenJDK 64-Bit Server VM (13.0.1+9, mixed mode, sharing, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# C [jna515356041985641679.tmp+0x12b85] ffi_prep_closure_loc+0x15
[OS:Red Hat Enterprise Linux Server release 7.8 (Maipo)
uname:Linux 3.10.0-1127.10.1.el7.x86_64 #1 SMP Tue May 26 15:05:43 EDT 2020 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 4096, NOFILE 65535, AS infinity, DATA infinity, FSIZE infinity
load average:0.08 0.03 0.05
.../...
It works like a charm on CentOS without doing anything.
For RHEL, I already fixed the stuff about JNA by adding ES_TMPDIR=/var/es-temp into /etc/sysconfig/elasticsearch as
Memory seems fine. this is a brand new VM. (no application logs into /var/logs)
Seems that this version is supposed to be supported
I tested with -Xms2g -Xmx2g, -Xms1g -Xmx1g, -Xms512m -Xmx512m but same error.
I don't get what is going wrong. My Next step is to test with another version 7 of elasticsearch.
After 1 day of struggling, I found the solution at https://discuss.elastic.co/t/elasticsearch-v7-6-2-failed-to-start-killed-by-sigabrt-on-rhel-7-7-urgent/231039/11 from Ivan_A_Carrazana_C
I put here a copy of the steps to perform:
Hi
If you are applying a security compliance in your RHEL installation you must change the >path of the TMP directory that will use elasticsearch as Java.
Uncomment at /etc/elasticsearch/jvm.options
-Djava.io.tmpdir=${ES_TMPDIR}
Add in /etc /sysconfig/elasticsearch
ES_TMPDIR=/usr/share/elasticsearch/tmp
Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group >are elasticsearch and the permissions are 0755
Lastly make sure that /dev/shm doesn't have the noexec attribute with command:
mount | grep tmpfs | grep '/dev/shm'
Expected result:
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
If you get output like these:
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,seclabel)
Add or modify in /etc/fstab the following line:
tmpfs /dev/shm tmpfs defaults,nodev,nosuid 0 0
I had the same problem and this worked for me. Hope i can help you
Seems to be known by elastic but not documented correctly. don't undertand why the tmpfs should in noexec. Would be good to have an JNA expert feedback about it.
For some reason, adding a TMPDIR var to /etc/sysconfig/elasticsearch worked (on 7.7.1) and pointing it to the same location as -Djava.io.tmpdir.
i.e.
TMPDIR="/usr/share/elasticsearch/tmp"
(in my case I actually used /var/lib/elasticsearch/tmp with 0755 permissions on it).
I can't say why, and it doesn't change the call string used if I look at 'ps -aef' . But just having -Djava.io.tmpdir wasn't enough.
This allowed me to get it to work without removing noexec on /tmp and /dev/shm.
I am new in ELK configuration.
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-16-04
I have configure in my local machine and it is work fine.
I want to forward my application logs file to elastic search using log-stash of file beats.
When I have configure all things working fine for system logs.
but I am not able to store my application log to elastic search.
Please help me.
This is my log file:
service.log
{"name":"service name", "hostname":"abc", "pid":4474, "userId":"123", "school_id":"123", "role":"student", "username":"mahi123", "serviceName":"loginService", "level":40, "msg":"successFully fetch trail log", "time":"2019-06-01T10:55:46.482Z","v":0}
Some troubleshooting steps to take care of when logs do not reach Elastisearch:
Check your log parsing configuration file(usually made with the extension .conf). Make sure it's having the right path to scan logs from, right set of filters etc. To see if this .conf file is actually working, one can try:
logstash -f <elasticsearch.conf file path> If this doesn't throw any error on console, that means you are good at this point and will have to move to next step.
Check if Kibana indices are getting created. Run
curl http://<hostipaddress or localhost>:9200/_cat/indices?v.
If yes, go to Kibana Management and create index patterns.
If not, check if your system has enough available memory to serve logstash and elastisearch. free -m would be helpful once you start logstash and elasticsearch services.
Many a times, I have seen people trying ELK setup on a machine which has insufficient RAM(4GB sounds good for a standalone setup).
Check your logstash and Elasticsearch services are up and running. If Elasticsearch is getting down or getting restarted during log parsing or indices creation, that's most probably due to lack of system resources.
-bash-4.2# systemctl status elasticsearch
�� elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-06-05 14:08:26 UTC; 1 weeks 0 days ago
Docs: http://www.elastic.co
Main PID: 1396 (java)
CGroup: /system.slice/elasticsearch.service
������1396 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMS...
Jun 05 14:08:26 cue-bldsvr4 systemd[1]: Started Elasticsearch.
Jun 05 14:08:26 cue-bldsvr4 systemd[1]: Starting Elasticsearch...
-bash-4.2# systemctl status logstash
�� logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-06-05 14:50:52 UTC; 1 weeks 0 days ago
Main PID: 4320 (java)
CGroup: /system.slice/logstash.service
������4320 /bin/java -Xms256m -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFrac...
Jun 05 14:50:52 cue-bldsvr4 systemd[1]: Started logstash.
Jun 05 14:50:52 cue-bldsvr4 systemd[1]: Starting logstash...
Jun 05 14:51:08 cue-bldsvr4 logstash[4320]: Sending Logstash's logs to /var/log/logstash which is now configur...rties
Hint: Some lines were ellipsized, use -l to show in full.
-bash-4.2#
I've been trying to start ES for hours and I can't seem to be able to do so.
The command sudo service elasticsearch status prints out :
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since ven. 2019-01-11 12:22:33 CET; 5min ago
Docs: http://www.elastic.co
Process: 16713 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=$PID_DIR/elasticsearch.pid -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.confi Main PID: 16713 (code=exited, status=1/FAILURE)
janv. 11 12:22:33 glamuse systemd[1]: Started Elasticsearch.
janv. 11 12:22:33 glamuse systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
janv. 11 12:22:33 glamuse systemd[1]: elasticsearch.service: Unit entered failed state.
janv. 11 12:22:33 glamuse systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
I've increased the memory and done all the fixes I could find on the internet, but I can't seem to figure out what's going on, there's not even a single log generated today... So I don't even have any trace of where the error could be.
I'm using ES version 1.7.2 (yes, it's old, but that shouldn't be a problem as it does work, and no I can't upgrade because my Elastica uses this version, anyways ...)
I'm using a vagrant machine, so it's a unix based system.
My config is as follow (removed all the useless comments) :
index.number_of_shards: 10
index.number_of_replicas: 1
bootstrap.mlockall: true
network.bind_host: 0
network.host: 0.0.0.0
indices.recovery.max_bytes_per_sec: 200mb
indices.store.throttle.max_bytes_per_sec : 200mb
script.engine.groovy.inline.search: on
script.engine.groovy.inline.aggs: on
script.engine.groovy.inline.update: on
index.query.bool.max_clause_count: 100000
I also have this conf :
ES_HEAP_SIZE=4g
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
START_DAEMON=true
ES_USER=elasticsearch
ES_GROUP=elasticsearch
LOG_DIR=/var/log/elasticsearch
DATA_DIR=/var/lib/elasticsearch
WORK_DIR=/tmp/elasticsearch
CONF_DIR=/etc/elasticsearch
CONF_FILE=/etc/elasticsearch/elasticsearch.yml
RESTART_ON_UPGRADE=true
Any idea how can I debug this?