The error I am receiving when I try to start up elasticsearch
-- Unit elasticsearch.service has begun starting up.
Oct 08 23:54:05 ElasticSearch logstash[1064]: [2020-10-08T23:54:05,137][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=
Oct 08 23:54:05 ElasticSearch logstash[1064]: [2020-10-08T23:54:05,138][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=
Oct 08 23:54:05 ElasticSearch kernel: [UFW BLOCK] IN=eth0 OUT= MAC=76:67:e9:46:24:b8:fe:00:00:00:01:01:08:00 SRC=79.124.62.110 DST=206.189.196.214 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=52316 PROTO=
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: output:
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: error:
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: Unrecognized VM option 'UseConcMarkSweepGC'
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: Error: Could not create the Java Virtual Machine.
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: Error: A fatal exception has occurred. Program will exit.
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
Oct 08 23:54:05 ElasticSearch systemd-entrypoint[14701]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
It looks a lot like this reported issue and this one.
In your jvm.options file, if you replace this
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
with this
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
it should work again.
Related
Logstash pipeline is not ingesting data into the elasticsearch index though the pipeline was running. This pipeline was deployed one
year back and it was running well since then. But on 24th May 2021, it stopped ingesting data. Restarting the logstash fixed the issue.
we have checked logstash logs but did find nothing there. Please see the below log.
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,503][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>http://localhost:9200/}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,619][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,623][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Aug 26 12:24:22 xyz.com systemd[1]: Stopping logstash...
Aug 26 12:24:22 xyz.com logstash[827]: [2020-08-26T12:24:22,665][WARN ][logstash.runner ] SIGTERM received. Shutting down.
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,864][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>33, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>25, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>26, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>27, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>28, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>29, "name"=>"[main]>worker4", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>31, "name"=>"[main]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>32, "name"=>"[main]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,866][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,852][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x2c879c55 run>"}
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,855][INFO ][logstash.runner ] Logstash shut down.
Aug 26 12:24:29 xyz.com systemd[1]: Stopped logstash.
May 24 08:11:46 xyz.com systemd[1]: Started logstash.
May 24 08:12:11 xyz.com logstash[19174]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 08:12:11 xyz.com logstash[19174]: 2021-05-24 08:12:11,794 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open0(Native Method)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
May 24 08:12:11 xyz.com logstash[19174]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,120][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,363][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Main process exited, code=killed, status=9/KILL
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Failed with result 'signal'.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
May 24 15:38:50 xyz.com systemd[1]: Stopped logstash.
May 24 15:38:50 xyz.com systemd[1]: Started logstash.
May 24 15:39:06 xyz.com logstash[25666]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 15:39:07 xyz.com logstash[25666]: 2021-05-24 15:39:07,101 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFo
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open0(Native Method)
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Server OS: Ubuntu 18.04 ELK Version: 7.11.1
Need your help to find out the exact reason.
did you try updating the bundles to newer editions
We have deployed 10 logstash config files last year. We started all the config files at once by using the folder where we have kept all the config files(as service). On 24th May, we found that, few of them are not pushing data in the elasticsearch, checked from Kibana Discover. Though rest of them were working fine. We checked the status of all 3 ELK components first (by systemctl status), found that all 3 services are running. We have then checked logs of logstash by using journalctl but did not find anything that caused the issue. We then stopped all the PIDs that were running for logstash and then started the logstash pipelines and that fixed the issue.
Please find below the portion of the logs from journalctl:
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,503][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>http://localhost:9200/}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,619][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Aug 26 12:08:37 xyz.com logstash[827]: [2020-08-26T12:08:37,623][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Aug 26 12:24:22 xyz.com systemd[1]: Stopping logstash...
Aug 26 12:24:22 xyz.com logstash[827]: [2020-08-26T12:24:22,665][WARN ][logstash.runner ] SIGTERM received. Shutting down.
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,864][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>33, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>25, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>26, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>27, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>28, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>29, "name"=>"[main]>worker4", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>30, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>31, "name"=>"[main]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>32, "name"=>"[main]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}]}}
Aug 26 12:24:27 xyz.com logstash[827]: [2020-08-26T12:24:27,866][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,852][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x2c879c55 run>"}
Aug 26 12:24:29 xyz.com logstash[827]: [2020-08-26T12:24:29,855][INFO ][logstash.runner ] Logstash shut down.
Aug 26 12:24:29 xyz.com systemd[1]: Stopped logstash.
May 24 08:11:46 xyz.com systemd[1]: Started logstash.
May 24 08:12:11 xyz.com logstash[19174]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 08:12:11 xyz.com logstash[19174]: 2021-05-24 08:12:11,794 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open0(Native Method)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
May 24 08:12:11 xyz.com logstash[19174]: at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
May 24 08:12:11 xyz.com logstash[19174]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,120][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>
May 24 08:12:25 xyz.com logstash[19174]: [2021-05-24T08:12:25,363][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Main process exited, code=killed, status=9/KILL
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Failed with result 'signal'.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 24 15:38:50 xyz.com systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
May 24 15:38:50 xyz.com systemd[1]: Stopped logstash.
May 24 15:38:50 xyz.com systemd[1]: Started logstash.
May 24 15:39:06 xyz.com logstash[25666]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
May 24 15:39:07 xyz.com logstash[25666]: 2021-05-24 15:39:07,101 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFo
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open0(Native Method)
May 24 15:39:07 xyz.com logstash[25666]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Server OS: Ubuntu 18.04
ELK Version: 7.11.1
Please help us to find the cause of the issue as we are struggling to locate the issue.
Focus at
java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
Check your file/folder permission (chmod: read, write, execute).
Use root account, check permission
ls -l /path/to/file
ls -l /var/log/logstash/logstash-plain.log
See more at https://askubuntu.com/a/528433/299516
Error i get is:
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
After running command systemctl status apache2.service i get :
apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset:
Drop-In: /lib/systemd/system/apache2.service.d
└─apache2-systemd.conf
Active: failed (Result: exit-code) since Fri 2020-05-08 13:27:50 IST; 2min 12
Process: 12058 ExecStart=/usr/sbin/apachectl start (code=exited, status=127)
May 08 13:27:50 kaushal systemd[1]: Starting The Apache HTTP Server...
May 08 13:27:50 kaushal apachectl[12058]: /usr/sbin/apachectl: 174: /usr/sbin/ap
May 08 13:27:50 kaushal apachectl[12058]: Action 'start' failed.
May 08 13:27:50 kaushal apachectl[12058]: The Apache error log may have more inf
May 08 13:27:50 kaushal systemd[1]: apache2.service: Control process exited, cod
May 08 13:27:50 kaushal systemd[1]: apache2.service: Failed with result 'exit-co
May 08 13:27:50 kaushal systemd[1]: Failed to start The Apache HTTP Server.
After running journalctl -xe i get:
May 08 13:27:50 kaushal systemd[1]: apache2.service: Failed with result 'exit-co
May 08 13:27:50 kaushal sudo[12024]: pam_unix(sudo:session): session closed for
May 08 13:27:50 kaushal systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit apache2.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit apache2.service has failed.
--
-- The result is RESULT.
May 08 13:27:53 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:27:54 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:27:55 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:27:56 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:28:21 kaushal /usr/lib/gdm3/gdm-x-session[2772]: (EE) client bug: time
May 08 13:28:26 kaushal kernel: psmouse serio4: Touchpad at isa0060/serio4/input
May 08 13:28:26 kaushal kernel: psmouse serio4: Touchpad at isa0060/serio4/input
May 08 13:28:36 kaushal kernel: psmouse serio4: Touchpad at isa0060/serio4/input
May 08 13:28:36 kaushal kernel: psmouse serio4: Touchpad at isa0060/serio4/input
May 08 13:29:53 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:29:54 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:29:55 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
May 08 13:29:56 kaushal kernel: [UFW BLOCK] IN=wlp3s0 OUT= MAC=00:08:ca:f0:27:04
lines 1328-1350/1350 (END)
Elasticsearch working with no issues on http://localhost:9200
And Operating system is Ubuntu 18.04
Here is the error log for Kibana
root#syed-MS-7B17:/var/log# journalctl -fu kibana.service
-- Logs begin at Sat 2020-01-04 18:30:58 IST. --
Apr 03 20:22:49 syed-MS-7B17 kibana[7165]: {"type":"log","#timestamp":"2020-04-03T14:52:49Z","tags":["fatal","root"],"pid":7165,"message":"{ Error: listen EADDRNOTAVAIL: address not available 7.0.0.1:5601\n at Server.setupListenHandle [as _listen2] (net.js:1263:19)\n at listenInCluster (net.js:1328:12)\n at GetAddrInfoReqWrap.doListen (net.js:1461:7)\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)\n code: 'EADDRNOTAVAIL',\n errno: 'EADDRNOTAVAIL',\n syscall: 'listen',\n address: '7.0.0.1',\n port: 5601 }"}
Apr 03 20:22:49 syed-MS-7B17 kibana[7165]: FATAL Error: listen EADDRNOTAVAIL: address not available 7.0.0.1:5601
Apr 03 20:22:50 syed-MS-7B17 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 03 20:22:50 syed-MS-7B17 systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: Stopped Kibana.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Start request repeated too quickly.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 03 20:22:53 syed-MS-7B17 systemd[1]: Failed to start Kibana.
I have resolved it myself after checking the /etc/hosts file
It was edited by mistake like below
7.0.0.1 localhost
Dear StackOverflow community,
I was running Kibana/Elasticsearch without a problem until installing a Kibana plugin. Then the service failed and I noticed that the problem is that Elasticsearch stopped. I tried several ways to fix it, and then even reinstalled all. But the problem still avoiding to launch Elasticsearch, even with a fresh installation.
Installation on Debian 9 using apt install.
systemctl start elasticsearch.service
results on:
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
[0.000s][error][logging] Error opening log file '/gc.log': Permission denied
Full log with journalctl -xe
-- Unit elasticsearch.service has begun starting up.
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"No living connections"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal kibana[576]: {"type":"log","#timestamp":"2020-02-07T13:09:06Z","tags":["warning","elasticsearch","admin"],"pid":576,"message":"No living connections"}
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: output:
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: [0.000s][error][logging] Error opening log file '/gc.log': Permission denied
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: [0.000s][error][logging] Initialization of output 'file=/var/log/elasticsearch/gc.log' using options 'filecount=32,filesize=64m' failed.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: error:
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Error: Could not create the Java Virtual Machine.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: Error: A fatal exception has occurred. Program will exit.
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Feb 07 14:09:06 Debian-911-stretch-64-minimal elasticsearch[2312]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)
Feb 07 14:09:06 Debian-911-stretch-64-minimal systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 07 14:09:06 Debian-911-stretch-64-minimal systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit elasticsearch.service has failed.
The mentioned gc.log file was not in that folder. And the permissions were:
drwxr-s--- 2 elasticsearch elasticsearch 4096 Jan 15 13:20 elasticsearch
I created the file and also played with permissions until having these:
-rwxrwxrwx 1 root elasticsearch 0 Feb 7 15:19 gc.log
...and even changed the ownership:
-rwxrwxrwx 1 root root 0 Feb 7 15:19 gc.log
But no success, I still having the same issue.
Thanks
Make sure you are running CMD as Administrator.
This error also happens if you are using docker & running the container as a different user. You have to add --group_add flag to docker command or set TAKE_FILE_OWNERSHIP environment variable as mentioned here
Using docker-compose:
user: 1007:1007
group_add:
- 0
Using docker:
--group-add 0
Firstly, I didn't know why gc.log file was not present. Have you changed the logs folder path or something? The gc.log path can be set in jvm.options file. By default ES logs and java garbage collection logs are fed into the logs folder inside $ES_HOME directory.
About user perspective, elastic search can't be run as root user. So from the ES directory details its showing you have an elasticsearch user created, and trying to run the cluster by that user.
The problem here can be solved by changing the permissions of files insdie the ES directory where all it belongs. Now the gc.log file is owned by root user and it cannot be accessed by the elasticsearch user.
Try this: sudo chown <user> <path/to/es/directory> -R
Here it becomes : sudo chown elasticsearch elasticsearch/ -R
If the issue still persists, check the jvm.options file whether its all configured correctly. Unless you change the -Xloggc:logs/gc.log option, the gc.log won't be pushing to /var/log.
Feb 09 17:09:02 server elasticsearch[2199]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Your log says, the option is given as file=/var/log/elasticsearch/gc.log. Correct any wrong configurations as per documentation : https://www.elastic.co/guide/en/elasticsearch/reference/master/jvm-options.html
sudo systemctl -l status elasticsearch.service
Returns this log:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: failed (Result: exit-code) since Sun 2020-02-09 17:09:02 CET; 2min 48s ago
Docs: http://www.elastic.co
Process: 2199 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2199 (code=exited, status=1/FAILURE)
Feb 09 17:09:02 server elasticsearch[2199]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Feb 09 17:09:02 server elasticsearch[2199]: Error: Could not create the Java Virtual Machine.
Feb 09 17:09:02 server elasticsearch[2199]: Error: A fatal exception has occurred. Program will exit.
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Feb 09 17:09:02 server elasticsearch[2199]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)
Feb 09 17:09:02 server systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 09 17:09:02 server systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 09 17:09:02 server systemd[1]: Failed to start Elasticsearch.
At this point I'm doing a fresh install. Not able to find the solution I need to continue working...