My elasticsearch index were working from past year and suddenly I got this error:
java.lang.IllegalStateException: Unable to access 'path.data' (/var/data/lib/elasticsearch)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:316)
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:256)
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:212)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:45)
Caused by: java.nio.file.AccessDeniedException: /var/data/lib
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
at java.nio.file.Files.createDirectories(Files.java:746)
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
... 7 more
I haven't changed any file permissions. They were working fine till now.
cd /var/data/lib/elasticsearch
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Nov 19 2020 elasticsearch
Did this happen to anyone? I stopped my Elasticsearch, and now I am not able to start it. Any help is appreciated.
Related
I am setting up hive after I setup hadoop on my personal machine. However after I install all the steps to configure hive I am still unable to use hive. I see the following error message:
hadoop#ub:/usr/local/hive/lib$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = d7574f3f-4ec1-4fac-bbbd-631c5f698e3c
Exception in thread "main" java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap')
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413)
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389)
at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
When I run jps I get the following:
hadoop#acharya-ub:/usr/local/hive/lib$ jps
4866 Jps
28644 SecondaryNameNode
28117 NameNode
28965 ResourceManager
29353 NodeManager
28349 DataNode
After searching over the internet, I found that running hiverser2 helped them but it hasn't helped me and I still get the same error message even after I start hiveserver2 service.
I am using java 11, hadoop-3.3.0 and apache-hive-3.1.2
I created symbolic links for the hadoop and hive locations:
lrwxrwxrwx 1 root root 23 Sep 1 22:39 hadoop -> /usr/local/hadoop-3.3.0
drwxr-xr-x 11 hadoop hadoop 4096 Sep 1 22:46 hadoop-3.3.0
drwxrwxr-x 10 hadoop hadoop 4096 Sep 1 23:02 apache-hive-3.1.2-bin
lrwxrwxrwx 1 root root 33 Sep 1 23:03 hive -> /usr/local/apache-hive-3.1.2-bin/
I then setup the following in my .bashrc file:
HADOOP_HOME=/usr/local/hadoop
HIVE_HOME=usr/local/hive
Can someone help please?
I'm using Microsoft Cognitive Services within a Quarkus application. Everything works fine locally... including in a local Docker environment.
However when I deploy this thing to AKS... it logs the following error:
f506c46e-0234-4dc6-a3c8-27d6949cd4b1-1: org.jboss.resteasy.spi.UnhandledException: java.lang.UnsatisfiedLinkError: 'void com.microsoft.cognitiveservices.speech.SpeechConfig.setTempDirectory(java.lang.String)'
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:503)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2442)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1476)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsatisfiedLinkError: 'void com.microsoft.cognitiveservices.speech.SpeechConfig.setTempDirectory(java.lang.String)'
at com.microsoft.cognitiveservices.speech.SpeechConfig.setTempDirectory(Native Method)
at com.microsoft.cognitiveservices.speech.SpeechConfig.<clinit>(SpeechConfig.java:77)
I have checked the content of the tmp directory within the pod... and the files are created there... could this be due to the standard OS image that is added as part of a Quarkus container image?
For reference:
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
ARG JAVA_PACKAGE=java-11-openjdk-headless
Content of PODs tmp:
/work # ls -ltr /tmp
total 84
drwxr-xr-x 2 root root 4096 Aug 3 12:10 hsperfdata_root
drwxrwxrwx 3 root root 4096 Aug 3 12:10 vertx-cache
-rw-r--r-- 1 root root 3839 Aug 3 12:20 m4j3821112471677595333.tmp
-rw-r--r-- 1 root root 68372 Aug 3 12:20 m4j3337239382403748465.tmp
drwx------ 2 root root 4096 Aug 3 12:20 speech-sdk-native-15500065335689334576
Azure dependency:
<dependency>
<groupId>com.microsoft.cognitiveservices.speech</groupId>
<artifactId>client-sdk</artifactId>
<version>1.18.0</version>
</dependency>
Not sure what is causing this error. Did anyone experience this behaviour before?
I updated the data.path entry in elasticsearch.yml to point at a new location. However, upon trying to restart Elastic, I now get the following error:
Starting elasticsearch: Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.data' (/etc/lib/stuff)
Likely root cause: java.nio.file.AccessDeniedException: /etc/lib
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:250)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:227)
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:203)
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:184)
at org.elasticsearch.bootstrap.Security.configure(Security.java:105)
at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
For the sake of debugging, I'm trying to move the data.path from the default of var/lib/elasticsearch to /var/lib/stuff.
As far as I can tell, the user, group, and permissions match across these two directories:
$ ls -l /var/lib/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Jan 11 10:50 elasticsearch
...
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Jan 13 10:54 stuff
...
So why does Elastic throw an AccessDenied Exception?
If I remove my custom data.path it goes back to working without issue. Am I missing something?
I have to load ElasticSearch plugins with the offline instructions but it looks like I am getting a permissions error even when I open up permissions to the files completely and running as root! I've removed and reinstalled as well but this error is consistent.
[root#ip-10-94-146-130 bin]# ./plugin install file:///home/jmblnvr/shield/license-2.1.0.zip
-> Installing from file:/home/jmblnvr/shield/license-2.1.0.zip...
Trying file:/home/jmblnvr/shield/license-2.1.0.zip ...
Downloading .DONE
Verifying file:/home/jmblnvr/shield/license-2.1.0.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed license into /usr/share/elasticsearch/plugins/license
[root#ip-10-94-146-130 bin]# service elasticsearch restart
Stopping elasticsearch: [FAILED]
Starting elasticsearch: Exception in thread "main" java.lang.IllegalStateException: Unable to initialize plugins
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/license/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:86)
at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:302)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:108)
at org.elasticsearch.node.Node.<init>(Node.java:148)
at org.elasticsearch.node.Node.<init>(Node.java:129)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
[FAILED]
From /var/log:
[root#ip-10-94-146-130 elasticsearch]# cat elasticsearch.log
[2015-12-15 11:44:24,525][INFO ][node ] [Valtorr] stopping ...
[2015-12-15 11:44:24,639][INFO ][node ] [Valtorr] stopped
[2015-12-15 11:44:24,641][INFO ][node ] [Valtorr] closing ...
[2015-12-15 11:44:24,647][INFO ][node ] [Valtorr] closed
[2015-12-15 11:44:26,097][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2015-12-15 11:44:26,314][INFO ][node ] [Agon] version[2.1.0], pid[21703], build[72cd1f1/2015-11-18T22:40:03Z]
[2015-12-15 11:44:26,314][INFO ][node ] [Agon] initializing ...
[2015-12-15 11:44:26,318][ERROR][bootstrap ] Exception
java.lang.IllegalStateException: Unable to initialize plugins
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:111)
at org.elasticsearch.node.Node.<init>(Node.java:148)
at org.elasticsearch.node.Node.<init>(Node.java:129)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/license/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:86)
at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:302)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:108)
... 6 more
I was having the same sort of exception but after installing and while loading x-pack plugin on Elasticsearch 5.2.1:
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/x-pack/plugin-descriptor.properties
In my case, I started by adding elasticsearch user to the root group:
usermod -aG root elasticsearch
Afterwards, it was only necessary to modify the ownership of the plugins folder:
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/plugins/
Thanks to Val, simple solution is after installing plugins you need to quickly change ownership/groups if you are using the RPM install of Elastic:
chown -R root:elasticsearch /usr/share/elasticsearch
I am creating a spring xd stream like below :
stream create foo --definition "jdbc --url=jdbc:mysql://localhost:3306/test --driverClassName=com.mysql.jdbc.Driver --query='select * from employee' |hdfs --inputType=text/plain --rollover=200" --deploy
I am able to get the data in hdfs..ButI am getting error in the logs like below..Any reason for this kind of error..
The output is here:
-rw-r--r-- 3 root hdfs 216 2015-10-15 08:45 /xd/foo/foo-34.txt
-rw-r--r-- 3 root hdfs 216 2015-10-15 08:45 /xd/foo/foo-35.txt
-rw-r--r-- 3 root hdfs 216 2015-10-15 08:45 /xd/foo/foo-36.txt
-rw-r--r-- 3 root hdfs 0 2015-10-15 08:45 /xd/foo/foo-37.txt.tmp
But the error is this in the log file
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204)
... 21 more
2015-10-15 08:42:04,839 1.1.2.RELEASE ERROR task-scheduler-9 handler.LoggingHandler - org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: Failed to validate a newly established connection.
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:245)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:457)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy182.call(Unknown Source)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Failed to validate a newly established connection.
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:802)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:617)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:186)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204)
... 21 more
I've encountered this issue as well at times. I opened a JIRA issue to look into this: https://jira.spring.io/browse/XD-3620
In the meantime you can work around this by specifying "--testOnBorrow=false" when creating the job or the stream.