I updated the data.path entry in elasticsearch.yml to point at a new location. However, upon trying to restart Elastic, I now get the following error:
Starting elasticsearch: Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.data' (/etc/lib/stuff)
Likely root cause: java.nio.file.AccessDeniedException: /etc/lib
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:250)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:227)
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:203)
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:184)
at org.elasticsearch.bootstrap.Security.configure(Security.java:105)
at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
For the sake of debugging, I'm trying to move the data.path from the default of var/lib/elasticsearch to /var/lib/stuff.
As far as I can tell, the user, group, and permissions match across these two directories:
$ ls -l /var/lib/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Jan 11 10:50 elasticsearch
...
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Jan 13 10:54 stuff
...
So why does Elastic throw an AccessDenied Exception?
If I remove my custom data.path it goes back to working without issue. Am I missing something?
Related
My elasticsearch index were working from past year and suddenly I got this error:
java.lang.IllegalStateException: Unable to access 'path.data' (/var/data/lib/elasticsearch)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:316)
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:256)
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:212)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:183)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:45)
Caused by: java.nio.file.AccessDeniedException: /var/data/lib
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
at java.nio.file.Files.createDirectories(Files.java:746)
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
... 7 more
I haven't changed any file permissions. They were working fine till now.
cd /var/data/lib/elasticsearch
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Nov 19 2020 elasticsearch
Did this happen to anyone? I stopped my Elasticsearch, and now I am not able to start it. Any help is appreciated.
I'm using Microsoft Cognitive Services within a Quarkus application. Everything works fine locally... including in a local Docker environment.
However when I deploy this thing to AKS... it logs the following error:
f506c46e-0234-4dc6-a3c8-27d6949cd4b1-1: org.jboss.resteasy.spi.UnhandledException: java.lang.UnsatisfiedLinkError: 'void com.microsoft.cognitiveservices.speech.SpeechConfig.setTempDirectory(java.lang.String)'
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:503)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2442)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1476)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.UnsatisfiedLinkError: 'void com.microsoft.cognitiveservices.speech.SpeechConfig.setTempDirectory(java.lang.String)'
at com.microsoft.cognitiveservices.speech.SpeechConfig.setTempDirectory(Native Method)
at com.microsoft.cognitiveservices.speech.SpeechConfig.<clinit>(SpeechConfig.java:77)
I have checked the content of the tmp directory within the pod... and the files are created there... could this be due to the standard OS image that is added as part of a Quarkus container image?
For reference:
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
ARG JAVA_PACKAGE=java-11-openjdk-headless
Content of PODs tmp:
/work # ls -ltr /tmp
total 84
drwxr-xr-x 2 root root 4096 Aug 3 12:10 hsperfdata_root
drwxrwxrwx 3 root root 4096 Aug 3 12:10 vertx-cache
-rw-r--r-- 1 root root 3839 Aug 3 12:20 m4j3821112471677595333.tmp
-rw-r--r-- 1 root root 68372 Aug 3 12:20 m4j3337239382403748465.tmp
drwx------ 2 root root 4096 Aug 3 12:20 speech-sdk-native-15500065335689334576
Azure dependency:
<dependency>
<groupId>com.microsoft.cognitiveservices.speech</groupId>
<artifactId>client-sdk</artifactId>
<version>1.18.0</version>
</dependency>
Not sure what is causing this error. Did anyone experience this behaviour before?
I'm trying to work out how to fix this ActiveMQ Artemis error.
Seems the occasional message is too big for SimpleString, and isn't sending, and it goes to the DLQ.
java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1366648 is greater than readableBytes=127646#ClientLargeMessageImpl[messageID=578576793, durable=true, address=AuthCorrespondence.sendmail,userID=7f72137c-c3a3-11eb-87f7-0242c0a8e003,properties=TypedProperties[__AMQ_CID=b3f70eb1-be3c-11eb-87f7-0242c0a8e003,_AMQ_LARGE_SIZE=127651,_AMQ_ROUTING_TYPE=1]]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:234)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:132)
at org.springframework.jms.support.destination.JmsDestinationAccessor.receiveFromConsumer(JmsDestinationAccessor.java:130)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:416)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:302)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1168)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1160)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1057)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1366648 is greater than readableBytes=127646
at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:183)
at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
at org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
at org.apache.activemq.artemis.reader.TextMessageUtil.readBodyText(TextMessageUtil.java:37)
at org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:112)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:228)
... 11 more
The most likely issue I can see is the similarity between readableBytes=127646 and _AMQ_LARGE_SIZE=127651.
From the docs, though, this _AMQ_LARGE_SIZE is the threshold for Large Messages, and it is supposed to be 2GB, and this message is what, 1.36MB?
What's going on?
EDIT:
[root#6dcbad102045 large-messages]# pwd
/opt/amq/broker/data/large-messages
[root#6dcbad102045 large-messages]# ls -l
total 13828
-rw-r--r-- 1 root root 6451200 Sep 14 2020 194154444.msg
-rw-r--r-- 1 root root 4198400 Nov 5 2020 266358970.msg
-rw-r--r-- 1 root root 1843200 Nov 13 2020 277265384.msg
-rw-r--r-- 1 root root 1433600 Apr 28 12:36 522483226.msg
-rw-r--r-- 1 root root 102400 Jun 2 15:07 578576791.msg
-rw-r--r-- 1 root root 127651 Jun 3 09:46 579961682.msg
I'm in Fuse/OSGi. 2.6.3.redhat-00015 for ActiveMQ Artemis JMS Client OSGi. 2.21.5 for camel-amqp. I can't work out what Artemis version it is. There's 1000+ successful deliveries. Just 6 fails.
The 2.6.3.redhat-00015 version corresponds to AMQ 7.2.3 which is quite old at this point. The current AMQ release is 7.8.1. I strongly recommend you upgrade as it's likely you're hitting a bug that's already been fixed.
You may be able to work around the issue by increasing the minimum large message size (e.g using minLargeMessageSize on core client URLs or amqpMinLargeMessageSize on your AMQP acceptor). For what it's worth, the stack-trace indicates that the core JMS client (i.e. not AMQP) is in use when the exception is thrown.
Lastly, it's worth noting that the default minimum large message size is 100 KB not 2 GB as explained in the documentation.
I am following this document https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options
to connect to my external hive metastore. My metastore version is 3.1.0 and followed the document.
docs.databricks.comdocs.databricks.com
External Apache Hive metastore — Databricks Documentation
Learn how to connect to external Apache Hive metastores in Databricks.
10:51
I have getting this error when trying to connect to external hive metastore
org/apache/hadoop/hive/conf/HiveConf when creating Hive client using classpath:
Please make sure that jars for your version of hive and hadoop are included in the paths passed to spark.sql.hive.metastore.jars
spark.sql.hive.metastore.jars=/databricks/hive_metastore_jars/*
When I do an ls on /databricks/hive_metastore_jars/, I can see all copied files
10:52
Do I need to copy any hive specific files and upload it in this folder?
I did exactly what was mentioned in the site
These are the contents of my hive_metastore_jars
total 56K
drwxr-xr-x 3 root root 4.0K Mar 24 05:06 1585025573715-0
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 d596a6ec-e105-4a6e-af95-df3feffc263d_resources
drwxr-xr-x 3 root root 4.0K Mar 24 05:06 repl
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 spark-2959157d-2018-441a-a7d3-d7cecb8a645f
drwxr-xr-x 4 root root 4.0K Mar 24 05:06 root
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 spark-30a72ee5-304c-432b-9c13-0439511fb0cd
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 spark-a19d167b-d571-4e58-a961-d7f6ced3d52f
-rwxr-xr-x 1 root root 5.5K Mar 24 05:06 _CleanRShell.r3763856699176668909resource.r
-rwxr-xr-x 1 root root 9.7K Mar 24 05:06 _dbutils.r9057087446822479911resource.r
-rwxr-xr-x 1 root root 301 Mar 24 05:06 _rServeScript.r1949348184439973964resource.r
-rwxr-xr-x 1 root root 1.5K Mar 24 05:06 _startR.sh5660449951005543051resource.r
Am I missing anything?
Strangely If I look into the cluster boot logs here is what I get
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionDriverName unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionURL unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionUserName unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionPassword unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property datanucleus.schema.autoCreateAll unknown - will be ignored
20/03/24 07:29:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
20/03/24 07:29:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
I have already set the above configurations and it shows in the logs as well
20/03/24 07:28:59 INFO SparkContext: Spark configuration:
spark.hadoop.javax.jdo.option.ConnectionDriverName=org.mariadb.jdbc.Driver
spark.hadoop.javax.jdo.option.ConnectionPassword=*********(redacted)
spark.hadoop.javax.jdo.option.ConnectionURL=*********(redacted)
spark.hadoop.javax.jdo.option.ConnectionUserName=*********(redacted)
Also version information is available in my hive metastore, I can connect to mysql and see it it shows
SCHEMA_VERSION : 3.1.0
VER_ID = 1
From the output, it looks like the jars are not copied to the "/databricks/hive_metastore_jars/" location. As mentioned in the documentation link you shared:
Set spark.sql.hive.metastore.jars set to maven
Restart the cluster with the above configuration and then check in the Spark driver logs for the message :
17/11/18 22:41:19 INFO IsolatedClientLoader: Downloaded metastore jars to <path>
From this location copy the jars to DBFS from the same cluster and then use an init script to copy the jars from DBFS to "/databricks/hive_metastore_jars/"
As I am using azure mysql there is one more step I need to perform
https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore
I am trying to installing CDH 4.6 in my cluster which is of 3 nodes.
One data node out of this 3 is not able to start at all.
Tried searching and solving this by all possible ways, but failed.
Please help me in solving this.
Below is the log.
5:49:10.708 PM FATAL org.apache.hadoop.hdfs.server.datanode.DataNode
Exception in secureMain
java.io.IOException: the path component: '/' is world-writable. Its permissions are 0777. Please fix this or select a different socket path.
at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)
at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:191)
at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:42)
at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:603)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:570)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:741)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:344)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1795)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1728)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1925)
5:49:10.723 PM INFO org.apache.hadoop.util.ExitUtil
Exiting with status 1
5:49:10.725 PM INFO org.apache.hadoop.hdfs.server.datanode.DataNode
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at xx.xx.xxx.xxxxx
Have you confirmed that your root filesystem is not set to 777 permissions?
This should be the correct permissions for root(/):
[root#server ~]# ls -Ald /
dr-xr-xr-x. 29 root root 4096 Feb 20 13:53 /
If you see this, then your root filesystems need to be chmod 555:
[root#server ~]# ls -Ald /
drwxrwxrwx. 29 root root 4096 Feb 20 13:53 /
Changing permission to 755 of root filesystem will resolve issue