I'm trying to run Flink 1.7.2 on ECS with Fargate. I've set up the state backend for my job to be RocksDB with a path=s3://...
In my Dockerfile my base image is 1.7.2-hadoop27-scala_2.11, and I run the following 2 commands:
RUN echo "fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider" >> "$FLINK_CONF_DIR/flink-conf.yaml"
RUN cp /opt/flink/opt/flink-s3-fs-hadoop-1.7.2.jar /opt/flink/lib/flink-s3-fs-hadoop-1.7.2.jar
Just like it says in
https://issues.apache.org/jira/browse/FLINK-8439
However I get the following exception:
Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider not found
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:592)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:556)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:256)
at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)
at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:407)
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)
... 17 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider not found
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2375)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2446)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:589)
... 28 more
Looking in the flink-s3-fs-hadoop-1.7.2.jar I see that the package of the class ContainerCredentialsProvider is actually org.apache.flink.fs.s3base.shaded.com.amazonaws.auth
I already tried:
Adding the aws-sdk-core jar to the lib, and setting the credentials provider to be just com.amazonaws.auth.ContainerCredentialsProvider (without the shading) but I get the problem mentioned in the issue link above
Setting the credentials provider to be org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.ContainerCredentialsProvider but then the code in S3FileSystemFactory.java prefixes it with org.apache.flink.fs.s3hadoop.shaded.
Any ideas here for finding the class?
The issue was resolved in one of the versions after.
I ran it on Flink 1.9.0 cluster with the following line:
RUN echo "fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider" >> "$FLINK_CONF_DIR/flink-conf.yaml"
and the class is found and works.
You can see in:
https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java
that the FLINK_SHADING_PREFIX is now correct
Related
My Grails 5.2 application fails to start on Heroku with this error:
Caused by: java.lang.IllegalArgumentException: More than one fragment with the name [org_apache_jasper] was found. This is not legal with relative ordering. See section 8.2.2 2c of the Servlet specification for details. Consider using absolute ordering. Duplicate fragments found in [[jar:file:/app/build/libs/OLRMain-1.3.0-plain.war!/WEB-INF/lib/tomcat-embed-jasper-9.0.68.jar, jar:file:/app/build/libs/OLRMain-1.3.0-plain.war!/WEB-INF/lib/webapp-runner-9.0.70.0.jar, jar:file:/app/build/libs/OLRMain-1.3.0-plain.war!/WEB-INF/lib/tomcat-jasper-9.0.70.jar]].
Here is the error with better formatting:
Caused by: java.lang.IllegalArgumentException: More than one fragment with the name [org_apache_jasper] was found.
Duplicate fragments found in [
[jar:file:/app/build/libs/OLRMain-1.3.0-plain.war!/WEB-INF/lib/tomcat-embed-jasper-9.0.68.jar,
jar:file:/app/build/libs/OLRMain-1.3.0-plain.war!/WEB-INF/lib/webapp-runner-9.0.70.0.jar,
jar:file:/app/build/libs/OLRMain-1.3.0-plain.war!/WEB-INF/lib/tomcat-jasper-9.0.70.jar]]
I understand the duplication error but don't know how to fix. I am deploying to Heroku.
Below is my Heroku procfile.
web: cd build; java $JAVA_OPTS -Dgrails.env=prod -jar ../build/server/webapp-runner-9.0.70.0.jar --expand-war --port $PORT libs/*.war
I've seen similar issues on SO such as "clearing" tomcat, and adding an entry to web.xml but don't know how to do this using Grails and Heroku.
I am working on backbase identity version 2022-08-LTS. Trying to use custom authenticator using link but when i execute the command java -cp ../custom-authenticator/target/custom-authenticator-2022.08-LTS-jar-with-dependencies.jar:backbase-identity-1.10.6.1-cr.0.jar io.quarkus.runner.GeneratedMain from backbase-identity module then got error Error: Could not find or load main class io.quarkus.runner.GeneratedMain Caused by: java.lang.ClassNotFoundException: io.quarkus.runner.GeneratedMain can any one help me on it
I find similar error on https://issues.apache.org/jira/browse/KYLIN-2511
env:
hadoop-2.7.1
hbase-1.3.2
apache-hive-2.1.1-bin
apache-kylin-1.6.0-hbase1.x-bin
I've tried copy all the hive libs to kylin, but get another ERROR.
org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/typeinfo/TypeInfo
The missing class should be in hive-exec-.jar; Check and debug the "bin/find-hive-dependency.sh" to see why it wasn't able to locate this jar from your server. You can manually add it to the "hive_exec_path" variable.
BTW, Kylin 1.6 is quite old, try to upgrade to a 2.x version.
Why you just try the method mentioned in https://issues.apache.org/jira/browse/KYLIN-2511. You'd better prepare the env according to the document of v16. It is better for using the latest version of Kylin. It has more feature and fixes some bugs.
I'm on an AWS EC2 VM (Ubuntu 14.04), willing to do some basics with Spark on RDDs from my S3 files. While running successfully this dirty command (not using sparkContext.hadoopConfiguration for the moment)
scala> val distFile = sc.textFile("s3n://< AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>#bucketname/folder1/folder2/file.csv")
I then get the following error when running a distFile.count()
java.lang.NoClassDefFoundError: org/jets3t/service/ServiceException
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.createDefaultStore(NativeS3FileSystem.java:334)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:324)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
...
...
Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
I have previously
defined an AWS IAM user with corresponding AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
added the export of both keys as env variables in .bashrc
built Spark 1.3.1 with SPARK_HADOOP_VERSION=2.6.0-cdh5.4.1 sbt/sbt assembly
installed and run hadoop 2.6-cdh5.4.1 (pseudo distributed)
Does it have to do with the syntax of the textFile("s3n// ...") ? I've tried others, including s3:// without success ...
Thank you
Include Jets3t jar to your class-path. Add proper compatible version with your current setting. You need ServiceException to be added to your class path.
I had the same problem. In spite it occurred at spark v2.1.0 with hadoop v2.7.2 environment, however I left it here because it would be the same cause. Here's what I've got.
A needed class was not found. This could be due to an error in your runpath. Missing class: org/jets3t/service/ServiceException
java.lang.NoClassDefFoundError: org/jets3t/service/ServiceException
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.createDefaultStore(NativeS3FileSystem.java:342)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:332)
at
...
...
Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
It was because of the classpath got the lower version of net.java.dev.jets3t:jets3t dependency than what org.apache.hadoop:hadoop-aws required.
I solved the problem after adding net.java.dev.jets3t:jets3t:0.9.0 in my build.sbt
You need to include hadoop-mapreduce-client jars in your CLASSPATH. In my case, I made my own distribution with these dependencies.
I put the following files in the lib folder:
hadoop-mapreduce-client-jobclient-2.6.0.jar
hadoop-mapreduce-client-hs-plugins-2.6.0.jar
hadoop-mapreduce-client-shuffle-2.6.0.jar
hadoop-mapreduce-client-jobclient-2.6.0-tests.jar
hadoop-mapreduce-client-common-2.6.0.jar
hadoop-mapreduce-client-app-2.6.0.jar
hadoop-mapreduce-client-hs-2.6.0.jar
hadoop-mapreduce-client-core-2.6.0.jar
I am trying to install twitter river plugin on elastic search. I have done it before on my mac without any issues, but this time I am trying to install on a Linux server and I keep getting this class not found exception. The exception comes when I use curl -XPUT command as shown in https://github.com/elasticsearch/elasticsearch-river-twitter/blob/master/README.md
Here's the exception I get :-
[2013-11-25 15:12:00,116][WARN ][river ] [Fasaud] failed to create river [twitter][my_twitter_river2]
org.elasticsearch.common.settings.NoClassSettingsException: Failed to load class with value [twitter]
at org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:87)
at org.elasticsearch.river.RiverModule.spawnModules(RiverModule.java:58)
at org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.java:44)
at org.elasticsearch.river.RiversService.createRiver(RiversService.java:135)
at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:271)
at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:265)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.ClassNotFoundException: twitter
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:73)
... 9 more
I have tried restarting, installing many many times and can't get it to work anyhow. I can also see that the jar files present in the plugin directories contain the class twitter but for some reason they're not available to elasticsearch execution.
This is a configuration and install issue.
Steps:
Check to see where elasticsearch is installed by running ps aux|grep elasticsearch (on linux) and that will tell which executable is running on that machine (look for a line that looks like this: (-Des.default.path.home=/usr/share/elasticsearch) or something similar.
If several elasticsearch instances running, kill them all and start the one you care about.
Ensure that only one instance of elasticsearch is running (if that is the intent) and, from step one above, go to the home path directory.
Ensure that the plugin is installed in this directory in elasticsearch/plugins dir.
Restart elasticsearch.
confirm your classpath contain a text file es-plugin.properties with content
plugin=your.package.YourRiverPlugin
missing of such a property file shows almost the same error.
for more info, please refer http://blog.trifork.com/2013/01/10/how-to-write-an-elasticsearch-river-plugin/