Jetty 9 Error: Suppressed: java.lang.Throwable: HttpInput failure - jetty-9

I am using the Jetty 9.4.15 server to deploy the war in a standalone mode with the Http2.
The number of concurrent streams configured are jetty.http2.maxConcurrentStreams=1000 instead of default 128. This is because I am getting too many concurrent streams error as the client is having more streams.
The application is working fine, except some sporadic errors as shown below in the server logs.
org.eclipse.jetty.io.EofException: Reset cancel_stream_error
at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:156) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:574) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:343) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:252) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:294) ~[?:?]
at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onReset(Parser.java:368) ~[?:?]
at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139) ~[?:?]
at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97) ~[?:?]
at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66) ~[?:?]
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194) ~[?:?]
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123) ~[?:?]
at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[?:?]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[?:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[?:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
Suppressed: java.lang.Throwable: HttpInput failure
at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:831) ~[?:?]
at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:323) ~[?:?]
at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:219) ~[?:?]
at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:156) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:574) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:343) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:252) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:294) ~[?:?]
at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onReset(Parser.java:368) ~[?:?]
at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139) ~[?:?]
at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97) ~[?:?]
at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66) ~[?:?]
at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194) ~[?:?]
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123) ~[?:?]
at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115) ~[?:?]
at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[?:?]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[?:?]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[?:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[?:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
`
The client logs shows the below error.
Client.Timeout exceeded while awaiting headers.
The client is using streaming instead of waiting for complete response. I am not sure why this error is occurring some times. The client timeout is configured as 30 Seconds, I checked the application logs, where the application processing logic never exceeded 2 seconds.
Could someone help me in finding the root cause for this issue?

Related

Wrong files are loading while starting elasticsearch server (installed OpenDistro in it)

Describe the bug
I have installed OpenDistro plugin using the following command and created the following config files manually. I have configured the elasticsearch.yml file with the created pem files.
Insall OpenDistro Plugin: sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install -b com.amazon.opendistroforelasticsearch:opendistro_security:1.11.0.0
Config files:
root-ca.pem
admin-key.pem
admin.pem
node-key.pem
node.pem
Configured in elasticsearch.yml,
######## Start OpenDistro for Elasticsearch Security Configuration ########
# WARNING: revise all the lines below before you go into production
opendistro_security.ssl.transport.pemcert_filepath: node.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: node.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
- O=Test,L=Chennai, C=IN
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
opendistro_security.system_indices.enabled: true
opendistro_security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
######## End OpenDistro for Elasticsearch Security Configuration ########
But I got the following error while starting the server,
Error:
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:700) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:642) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:328) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:691) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:642) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:328) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more
**Caused by: org.elasticsearch.ElasticsearchException: Unable to read /usr/share/elasticsearch/config/esnode.pem (/usr/share/elasticsearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath**
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:929) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:226) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:350) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:247) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.<init>(DefaultOpenDistroSecurityKeyStore.java:168) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.<init>(OpenDistroSecuritySSLPlugin.java:210) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.<init>(OpenDistroSecurityPlugin.java:244) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:691) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:642) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:328) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more
uncaught exception in thread [main]
Why it's expecting esnode.pem instead of using node.pem?
Kindly provide your thoughts on this

I would suggest using the free security that is included in the default distribution, as this approach isn't supported by Elasticsearch

Elasticsearch is not starting for Search Guard error

I'm trying to configure Search Guard admin. here is my elasticsearch.yml configurations:
searchguard.ssl.transport.pemcert_filepath: /home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem
searchguard.ssl.transport.pemkey_filepath: /home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.key.pem
searchguard.ssl.transport.pemkey_password: changeit
searchguard.ssl.transport.pemtrustedcas_filepath: /home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/ca/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=Test,C=DE
After this configurations, I'm unable to start my elasticsearch. Here is the full error from my "/var/log/elasticsearch/my-application.log" file.
[2020-07-05T04:51:39,896][INFO ][o.e.n.Node ] [node-1] version[7.8.0], pid[2328], build[default/deb/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/4.9.0-12-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-07-05T04:51:39,907][INFO ][o.e.n.Node ] [node-1] JVM home [/usr/share/elasticsearch/jdk]
[2020-07-05T04:51:39,908][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-3499631201687762622, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-07-05T04:51:44,548][INFO ][c.f.s.s.SignalsPainlessExtension] [node-1] Loaded script whitelist: org.elasticsearch.painless.spi.Whitelist#3a5e2525
[2020-07-05T04:51:44,571][INFO ][c.f.s.SearchGuardPlugin ] [node-1] ES Config path is /etc/elasticsearch
[2020-07-05T04:51:44,877][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [node-1] JVM supports TLSv1.3
[2020-07-05T04:51:44,884][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [node-1] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2020-07-05T04:51:44,889][ERROR][o.e.b.Bootstrap ] [node-1] Exception
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
... 15 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
at sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) ~[?:?]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.nio.file.Files.readAttributes(Files.java:1843) ~[?:?]
at java.nio.file.Files.isDirectory(Files.java:2314) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:827) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:223) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:343) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:164) ~[?:?]
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:199) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:227) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
... 15 more
[2020-07-05T04:51:44,908][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
... 6 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
at sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) ~[?:?]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.nio.file.Files.readAttributes(Files.java:1843) ~[?:?]
at java.nio.file.Files.isDirectory(Files.java:2314) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:827) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:223) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:343) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:164) ~[?:?]
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:199) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:227) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
... 6 more
I'm following this and this documentation.
what is wrong here? why elasticsearch is not starting?

Judging from the error message it seems the location of your certificates is:
/home/rsa-key-20200528/search-guard-ssl-master/example-pki-scripts/node-0.crt.pem
However, the docs state that all certificates
must be under the config/ directory, specified using a relative path
(mandatory)
(https://docs.search-guard.com/latest/configuring-tls#using-x509-pem-certificates-and-pkcs-8-keys)
Please try moving the certificates and key(s) in the config directory.

elasticsearch user dictionary

I want to use the symbol '#' as a user dictionary with elasticsearch
However, setting the character "C #" in the user dictionary results in an error
ES version 5.6
I am using the Kuromoji plugin
user dictionary
C#,C#,シーシャープ,カスタム名詞
error
java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.lucene.analysis.ja.dict.UserDictionary$1.compare(UserDictionary.java:94) ~[?:?]
at org.apache.lucene.analysis.ja.dict.UserDictionary$1.compare(UserDictionary.java:91) ~[?:?]
at java.util.TimSort.binarySort(TimSort.java:296) ~[?:1.8.0_171]
at java.util.TimSort.sort(TimSort.java:221) ~[?:1.8.0_171]
at java.util.Arrays.sort(Arrays.java:1512) ~[?:1.8.0_171]
at java.util.ArrayList.sort(ArrayList.java:1462) ~[?:1.8.0_171]
at java.util.Collections.sort(Collections.java:175) ~[?:1.8.0_171]
at org.apache.lucene.analysis.ja.dict.UserDictionary.<init>(UserDictionary.java:91) ~[?:?]
at org.apache.lucene.analysis.ja.dict.UserDictionary.open(UserDictionary.java:81) ~[?:?]
at org.elasticsearch.index.analysis.KuromojiTokenizerFactory.getUserDictionary(KuromojiTokenizerFactory.java:65) ~[?:?]
at org.elasticsearch.index.analysis.KuromojiTokenizerFactory.<init>(KuromojiTokenizerFactory.java:52) ~[?:?]
at org.elasticsearch.index.analysis.AnalysisRegistry.buildMapping(AnalysisRegistry.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.index.analysis.AnalysisRegistry.buildTokenizerFactories(AnalysisRegistry.java:176) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.index.analysis.AnalysisRegistry.build(AnalysisRegistry.java:154) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.index.IndexService.<init>(IndexService.java:145) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.index.IndexModule.newIndexService(IndexModule.java:363) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:448) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:413) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:378) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Exception thrown when starting Elastic search with X-pack

I have elasticsearch 5.2.0 with x-pack. I am in OS X Sierra and I downloaded the binaries from the official website
An exception shows up everytime I try to start elasticsearch. The Warning shown is the following:
[2017-02-25T11:30:06,354][WARN ][o.e.d.z.UnicastZenPing ] [k9ErfRE] [1] failed send ping to {#zen_unicast_127.0.0.1_0#}{Xruys5FnTRaRKgOnhThjEg}{127.0.0.1}{127.0.0.1:9300}
java.lang.IllegalStateException: handshake failed with {#zen_unicast_127.0.0.1_0#}{Xruys5FnTRaRKgOnhThjEg}{127.0.0.1}{127.0.0.1:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:364) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:393) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:500) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: org.elasticsearch.transport.RemoteTransportException: [Dc1gCWm][127.0.0.1:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [internal:transport/handshake]
at org.elasticsearch.xpack.security.support.Exceptions.authenticationError(Exceptions.java:39) ~[?:?]
at org.elasticsearch.xpack.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:74) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:483) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$13(AuthenticationService.java:315) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:320) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:247) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$5(AuthenticationService.java:223) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:236) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:184) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:201) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:213) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:180) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:142) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:114) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:142) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:296) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1488) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1445) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1329) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
Do you know what is it and why does it appear and how to make it stop?
N.B.: Elasticsearch correctly start after this warning, and everything seems to work fine (for example in querying it or adding indexes and elements to them), but it bothers me that warning and I am afraid that it is a sign for something worse.
Thank you in advance

Looks like an issue with Zen Discovery. Maybe you should increase the ping timeout. More info here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html

You must read the latest documentation from https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#installing-xpack
To use X-Pack you need:
Elasticsearch 5.3.0 - Installing Elasticsearch
Kibana 5.3.0 - Getting Kibana Up and Running
after all installed on u're machine
let's surf this documentation: https://www.elastic.co/guide/en/x-pack/current/security-getting-started.html

installing x-pack stops elasticsearch from running

Centos VM guest on Win10 host. Attempting to follow installation guide for x-pack (and step by step from the training course). ES and K running fine, but once x-pack plugin installed, kibana will start (and complains ES is missing - that's fine), then elastic search starts and immediately fails. It looks like the same error as 3, but in a local VM, not a production setting. Here's the tail of the log:
[2017-01-31T16:52:03,191][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.1.1.jar:5.1.1]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
... 14 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_111]
at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
... 14 more
[2017-01-31T16:52:03,215][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elkmaster1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: AccessDeniedException[/etc/elasticsearch/x-pack/system_key];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_111]
at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
... 6 more

try it out:
open config/elasticsearch.yml file
change xpack configuration line
xpack.security.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false

Well since the log contains
Caused by: java.nio.file.AccessDeniedException:
The problem is possibly the user permissions. Who is the owner of this directory?
/etc/elasticsearch/x-pack
What does ls -l /etc/elasticsearch/x-pack give you for an output?
If you have sudo permissions or have root access, you might just change the ownership of this directory.

The problem is with the user accounts you used to install elasitcsearch and x-pack. Make sure you used the same account and have the right permissions to do so!
You might want to grant access using chown.

Resources