I am tearing my hair trying to figure why ElasticSearch is not starting. These are my first days with ES stack, so I am completely helpless.
I am running
sudo systemctrl start elasticsearch
and get error saying that
Job for elasticsearch.service failed because a fatal signal was
delivered to the control process. See "systemctl status
elasticsearch.service" and "journalctl -xe" for details.
When I try to run journalctl -xe, I see lines of super-unclear messages for me:
Apr 21 19:16:24 my-pc-15IKB kernel: br-6f050fa6218c: port 2(veth19babe9) entered blocking state
Apr 21 19:16:24 my-pc-15IKB kernel: br-6f050fa6218c: port 2(veth19babe9) entered disabled state
Apr 21 19:16:24 my-pc-15IKB kernel: device veth19babe9 entered promiscuous mode
Apr 21 19:16:24 my-pc-15IKB charon[1828]: 10[KNL] interface veth19babe9 activated
Apr 21 19:16:24 my-pc-15IKB NetworkManager[757]: <info> [1619021784.2920] manager: (veth8fd9f3d): new Veth device (/org/freedesktop/NetworkManager/Devices/423)
Apr 21 19:16:24 my-pc-15IKB systemd-udevd[39880]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 21 19:16:24 my-pc-15IKB NetworkManager[757]: <info> [1619021784.2934] manager: (veth19babe9): new Veth device (/org/freedesktop/NetworkManager/Devices/424)
Apr 21 19:16:24 my-pc-15IKB kernel: br-6f050fa6218c: port 2(veth19babe9) entered blocking state
Apr 21 19:16:24 my-pc-15IKB kernel: br-6f050fa6218c: port 2(veth19babe9) entered forwarding state
Apr 21 19:16:24 my-pc-15IKB systemd-udevd[39880]: Using default interface naming scheme 'v245'.
Apr 21 19:16:24 my-pc-15IKB systemd-udevd[39880]: veth19babe9: Could not generate persistent MAC: No data available
Apr 21 19:16:24 my-pc-15IKB systemd-udevd[39877]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 21 19:16:24 my-pc-15IKB dockerd[1804]: time="2021-04-21T19:16:24.298236681+03:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using def>
Apr 21 19:16:24 my-pc-15IKB dockerd[1804]: time="2021-04-21T19:16:24.298287953+03:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 200>
Apr 21 19:16:24 my-pc-15IKB systemd-udevd[39877]: Using default interface naming scheme 'v245'.
Apr 21 19:16:24 my-pc-15IKB systemd-udevd[39877]: veth8fd9f3d: Could not generate persistent MAC: No data available
Apr 21 19:16:24 my-pc-15IKB containerd[851]: time="2021-04-21T19:16:24.328926349+03:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.cont>
Apr 21 19:16:24 my-pc-15IKB charon[1828]: 11[KNL] interface veth8fd9f3d deleted
Apr 21 19:16:24 my-pc-15IKB kernel: eth0: renamed from veth8fd9f3d
Apr 21 19:16:24 my-pc-15IKB kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth19babe9: link becomes ready
Apr 21 19:16:24 my-pc-15IKB NetworkManager[757]: <info> [1619021784.8637] device (veth19babe9): carrier: link connected
Apr 21 19:16:24 my-pc-15IKB gnome-shell[5088]: Removing a network device that was not added
Apr 21 19:16:26 my-pc-15IKB avahi-daemon[748]: Joining mDNS multicast group on interface veth19babe9.IPv6 with address fe80::a0c2:a3ff:feb8:587a.
Apr 21 19:16:26 my-pc-15IKB charon[1828]: 12[KNL] fe80::a0c2:a3ff:feb8:587a appeared on veth19babe9
Apr 21 19:16:26 my-pc-15IKB avahi-daemon[748]: New relevant interface veth19babe9.IPv6 for mDNS.
Apr 21 19:16:26 my-pc-15IKB avahi-daemon[748]: Registering new address record for fe80::a0c2:a3ff:feb8:587a on veth19babe9.*.
My .yml file looks this:
cluster.name: petlon-app
node.name: my-app-node
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
discovery.type: single-node
Could you please give a hint why does that mean ?
UPDATE:
The output of sudo journalctl -u elasticsearch.service
-- Logs begin at Thu 2021-03-11 13:10:55 MSK, end at Wed 2021-04-21 21:11:37 MSK. --
мар 16 21:23:20 my-pc systemd[1]: Starting Elasticsearch...
мар 16 21:23:41 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
мар 16 21:23:41 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
мар 16 21:23:41 my-pc systemd[1]: Failed to start Elasticsearch.
-- Reboot --
апр 11 15:55:31 my-pc systemd[1]: Starting Elasticsearch...
апр 11 15:55:46 my-pc systemd[1]: Started Elasticsearch.
апр 11 15:56:41 my-pc systemd[1]: Stopping Elasticsearch...
апр 11 15:56:41 my-pc systemd[1]: elasticsearch.service: Succeeded.
апр 11 15:56:41 my-pc systemd[1]: Stopped Elasticsearch.
апр 11 15:56:41 my-pc systemd[1]: Starting Elasticsearch...
апр 11 15:57:01 my-pc systemd[1]: Started Elasticsearch.
апр 11 16:11:06 my-pc systemd[1]: Stopping Elasticsearch...
апр 11 16:11:07 my-pc systemd[1]: elasticsearch.service: Succeeded.
апр 11 16:11:07 my-pc systemd[1]: Stopped Elasticsearch.
-- Reboot --
апр 11 16:12:53 my-pc systemd[1]: Starting Elasticsearch...
апр 11 16:13:13 my-pc systemd[1]: Started Elasticsearch.
апр 11 18:51:08 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 18:51:08 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:31:42 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:31:47 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:31:47 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:31:47 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:32:14 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:32:16 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:32:16 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:32:16 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:35:33 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:35:37 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:35:37 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:35:37 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:37:53 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:37:57 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:37:57 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:37:57 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:38:02 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:38:06 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:38:06 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:38:06 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:41:57 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:42:00 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:42:00 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:42:00 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:46:53 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:46:59 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:46:59 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:46:59 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:49:00 my-pc systemd[1]: Starting Elasticsearch...
апр 11 21:49:03 my-pc systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
апр 11 21:49:03 my-pc systemd[1]: elasticsearch.service: Failed with result 'signal'.
апр 11 21:49:03 my-pc systemd[1]: Failed to start Elasticsearch.
апр 11 21:49:32 my-pc systemd[1]: Starting Elasticsearch...
Elasticsearch working with no issues on http://localhost:9200
And Operating system is Ubuntu 18.04
Here is the error log for Kibana
root#syed-MS-7B17:/var/log# journalctl -fu kibana.service
-- Logs begin at Sat 2020-01-04 18:30:58 IST. --
Apr 03 20:22:49 syed-MS-7B17 kibana[7165]: {"type":"log","#timestamp":"2020-04-03T14:52:49Z","tags":["fatal","root"],"pid":7165,"message":"{ Error: listen EADDRNOTAVAIL: address not available 7.0.0.1:5601\n at Server.setupListenHandle [as _listen2] (net.js:1263:19)\n at listenInCluster (net.js:1328:12)\n at GetAddrInfoReqWrap.doListen (net.js:1461:7)\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)\n code: 'EADDRNOTAVAIL',\n errno: 'EADDRNOTAVAIL',\n syscall: 'listen',\n address: '7.0.0.1',\n port: 5601 }"}
Apr 03 20:22:49 syed-MS-7B17 kibana[7165]: FATAL Error: listen EADDRNOTAVAIL: address not available 7.0.0.1:5601
Apr 03 20:22:50 syed-MS-7B17 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 03 20:22:50 syed-MS-7B17 systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: Stopped Kibana.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Start request repeated too quickly.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: Failed to start Kibana.
I have resolved it myself after checking the /etc/hosts file
It was edited by mistake like below
7.0.0.1 localhost
Dear StackOverflow community,
I was running Kibana/Elasticsearch without a problem until installing a Kibana plugin. Then the service failed and I noticed that the problem is that Elasticsearch stopped. I tried several ways to fix it, and then even reinstalled all. But the problem still avoiding to launch Elasticsearch, even with a fresh installation.
Installation on Debian 9 using apt install.
systemctl start elasticsearch.service
results on:
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
[0.000s][error][logging] Error opening log file '/gc.log': Permission denied
Full log with journalctl -xe
-- Unit elasticsearch.service has begun starting up.
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"No living connections"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"No living connections"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: output:
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: [0.000s][error][logging] Error opening log file '/gc.log': Permission denied
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: [0.000s][error][logging] Initialization of output 'file=/var/log/elasticsearch/gc.log' using options 'filecount=32,filesize=64m' failed.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: error:
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Error: Could not create the Java Virtual Machine.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Error: A fatal exception has occurred. Program will exit.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)
Feb 07 14:09:06 Debian-911-stretch-64-minimal systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 07 14:09:06 Debian-911-stretch-64-minimal systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit elasticsearch.service has failed.
The mentioned gc.log file was not in that folder. And the permissions were:
drwxr-s--- 2 elasticsearch elasticsearch 4096 Jan 15 13:20 elasticsearch
I created the file and also played with permissions until having these:
-rwxrwxrwx 1 root elasticsearch 0 Feb 7 15:19 gc.log
...and even changed the ownership:
-rwxrwxrwx 1 root root 0 Feb 7 15:19 gc.log
But no success, I still having the same issue.
Thanks
Make sure you are running CMD as Administrator.
This error also happens if you are using docker & running the container as a different user. You have to add --group_add flag to docker command or set TAKE_FILE_OWNERSHIP environment variable as mentioned here
Using docker-compose:
user: 1007:1007
group_add:
- 0
Using docker:
--group-add 0
Firstly, I didn't know why gc.log file was not present. Have you changed the logs folder path or something? The gc.log path can be set in jvm.options file. By default ES logs and java garbage collection logs are fed into the logs folder inside $ES_HOME directory.
About user perspective, elastic search can't be run as root user. So from the ES directory details its showing you have an elasticsearch user created, and trying to run the cluster by that user.
The problem here can be solved by changing the permissions of files insdie the ES directory where all it belongs. Now the gc.log file is owned by root user and it cannot be accessed by the elasticsearch user.
Try this: sudo chown <user> <path/to/es/directory> -R
Here it becomes : sudo chown elasticsearch elasticsearch/ -R
If the issue still persists, check the jvm.options file whether its all configured correctly. Unless you change the -Xloggc:logs/gc.log option, the gc.log won't be pushing to /var/log.
Feb 09 17:09:02 server elasticsearch[2199]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Your log says, the option is given as file=/var/log/elasticsearch/gc.log. Correct any wrong configurations as per documentation : https://www.elastic.co/guide/en/elasticsearch/reference/master/jvm-options.html
sudo systemctl -l status elasticsearch.service
Returns this log:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: failed (Result: exit-code) since Sun 2020-02-09 17:09:02 CET; 2min 48s ago
Docs: http://www.elastic.co
Process: 2199 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2199 (code=exited, status=1/FAILURE)
Feb 09 17:09:02 server elasticsearch[2199]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Feb 09 17:09:02 server elasticsearch[2199]: Error: Could not create the Java Virtual Machine.
Feb 09 17:09:02 server elasticsearch[2199]: Error: A fatal exception has occurred. Program will exit.
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)
Feb 09 17:09:02 server systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 09 17:09:02 server systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 09 17:09:02 server systemd[1]: Failed to start Elasticsearch.
At this point I'm doing a fresh install. Not able to find the solution I need to continue working...
I've got a very simple bash script called test.
#!/bin/bash
while true
do
currdate=$(date +%d/%m/%Y-%H:%M:%S)
echo -e "\n$currdate ..."
sleep 1
done
Running this from the command line works exactly as expected.
I've compiled this with SHC has follows:
shc -r -U -f test.sh -o test
Running this from the command line works as expected.
I've setup to run this using systemctl, added my config file to /etc/init.d and this references test
If test is my un complied bash script, then systemclt start test works fine.
If I swap test and replace it with the complied version, then it fails.
systemctl status shows:
test.service - LSB: Test
Loaded: loaded (/etc/init.d/test; generated; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-02-18 15:38:05 GMT; 6s ago
Docs: man:systemd-sysv-generator(8)
Process: 18089 ExecStart=/etc/init.d/test start (code=exited, status=7)
Tasks: 2
CGroup: /system.slice/test.service
├─18096 /home/rocket/test -c >
└─18142 sleep 1
Feb 18 15:38:05 linux-c0a9 systemd[1]: test.service: Control process exited, code=exited status=7
Feb 18 15:38:05 linux-c0a9 systemd[1]: Failed to start LSB: Test.
Feb 18 15:38:05 linux-c0a9 systemd[1]: test.service: Unit entered failed state.
Feb 18 15:38:05 linux-c0a9 systemd[1]: test.service: Failed with result 'exit-code'.
journalctl shows:
Feb 18 15:38:05 linux-c0a9 systemd[1]: Starting LSB: Test...
Feb 18 15:38:05 linux-c0a9 test[18089]: Starting test
Feb 18 15:38:05 linux-c0a9 test[18089]: 18/02/2020-15:38:05 ...
Feb 18 15:38:05 linux-c0a9 test[18089]: ..failed
Feb 18 15:38:05 linux-c0a9 systemd[1]: test.service: Control process exited, code=exited status=7
Feb 18 15:38:05 linux-c0a9 systemd[1]: Failed to start LSB: Test.
Feb 18 15:38:05 linux-c0a9 systemd[1]: test.service: Unit entered failed state.
Feb 18 15:38:05 linux-c0a9 systemd[1]: test.service: Failed with result 'exit-code'.
Does anyone know why this fails and how to stop it failing ?
I'm only using SHC to stop users editing the script, not for any security etc.
Thanks
I think I may have solved this.
My script in /etc/init.d was calling /home/rocket/test
When that is a bash file it's fine, when it's compiled it stops working.
To get it to work I changed test to always be a bash file that calls the compiled test which is in in /home/rocket/bin/test
user#user-desktop:~$ sudo service apache2 restart
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
user#user-desktop:~$ systemctl status apache2.service
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/apache2.service.d
└─apache2-systemd.conf
Active: failed (Result: exit-code) since Tue 2018-09-18 16:45:15 IST; 7s ago
Process: 12099 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
Sep 18 16:45:14 user-desktop systemd[1]: Starting The Apache HTTP Server...
Sep 18 16:45:14 user-desktop apachectl[12099]: apache2: Syntax error on line 225 of /etc/apache2/apache2.conf: Syntax error on line
14 of /etc
Sep 18 16:45:15 user-desktop apachectl[12099]: Action 'start' failed.
Sep 18 16:45:15 user-desktop apachectl[12099]: The Apache error log may have more information.
Sep 18 16:45:15 user-desktop systemd[1]: apache2.service: Control process exited, code=exited status=1
Sep 18 16:45:15 user-desktop systemd[1]: apache2.service: Failed with result 'exit-code'.
Sep 18 16:45:15 user-desktop systemd[1]: Failed to start The Apache HTTP Server.