I am trying to run a spring boot embedded web container application that needs to talk to hdfs. I have included the spring data hadoop in my pom.xml but when I run the application on the cluster i get this error
Caused by: java.io.FileNotFoundException: JAR entry lib/hadoop-core-1.2.1.jar!/core-default.xml not found in hdfspub-0.0.1-SNAPSHOT.jar
at .jar.JarURLConnec...ction.java:122)
at JarURLConnec...ction.java:132)
at com.sun.org.apache.xerces.internal.impl.XMLEntityM anager.setupCurrentEntity(XMLEntityManager.java:65 1)
at com.sun.org.apache.xerces.internal.impl.XMLVersion Detector.determineDocVersion(XMLVersionDetector.ja va:186)
at com.sun.org.apache.xerces.internal.parsers.XML11Co nfiguration.parse(XML11Configuration.java:772)
at com.sun.org.apache.xerces.internal.parsers.XML11Co nfiguration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLPars er.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.DOMPars er.parse(DOMParser.java:232)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBu ilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBu ilder.java:180)
at org.apache.hadoop.conf.Configuration.loadResource( Configuration.java:1156)
Even thought if I look at my jar I can see that hadoop-core-1.2.1.jar is there
$ unzip -l hdfspub-0.0.1-SNAPSHOT.jar | grep hadoop-core-1.2.1.jar
4203713 10-25-2013 21:04 lib/hadoop-core-1.2.1.jar
Any help would be appreciated
I had same problem, solution is to add hadoop.core to your pom/build.gradle but you have to exclude from spring.data.hadoop and apache.hadoop groups:
tomcat
org.mortbay.jetty
javax.servlet
my build.gradle:
compile('org.apache.hadoop:hadoop-core:1.2.1'){
exclude group: 'org.mortbay.jetty'
exclude group: 'javax.servlet'
exclude group: 'tomcat'
}
compile('org.springframework.data:spring-data-hadoop:1.0.2.RELEASE') {
exclude group: 'org.mortbay.jetty'
exclude group: 'javax.servlet'
exclude group: 'tomcat'
}
Related
I have a project build with Gradle, it's actually a Vaadin project, with a servlet where I use Jetty.
At the startup (i.e gradle run) I have a lot of different warning message from AnnotationParser about duplication of classes. I copy only one because the log is quite verbose:
[INFO ] 11:22:50.375 org.eclipse.jetty.server.Server.doStart() - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 13.0.2+8
[WARN ] 11:22:50.777 org.eclipse.jetty.annotations.AnnotationParser.addParsedClass() - javax.websocket.ClientEndpoint scanned from multiple locations: jar:file:///Users/fox/.gradle/caches/modules-2/files-2.1/javax.websocket/javax.websocket-api/1.0/fc843b649d4a1dcb0497669d262befa3918c7ba8/javax.websocket-api-1.0.jar!/javax/websocket/ClientEndpoint.class, jar:file:///Users/fox/.gradle/caches/modules-2/files-2.1/javax.websocket/javax.websocket-client-api/1.0/afcf19e889d8725576811f8d47ab6c65d9dcbd78/javax.websocket-client-api-1.0.jar!/javax/websocket/ClientEndpoint.class
In this line AnnotationParser warns me that the ClientEndpoint.class is present in two jar, the javax.websocket-api-1.0.jar and the javax.websocket-client-api-1.0.jar.
With the command gradle dependencies I could see:
...
+--- org.eclipse.jetty.websocket:javax-websocket-client-impl:9.4.31.v20200723
| +--- org.eclipse.jetty.websocket:websocket-client:9.4.31.v20200723 (*)
| \--- javax.websocket:javax.websocket-client-api:1.0
+--- org.eclipse.jetty.websocket:websocket-server:9.4.31.v20200723 (*)
\--- javax.websocket:javax.websocket-api:1.0
In my gradle.build I only have:
dependencies {
// Vaadin
implementation enforcedPlatform('com.vaadin:vaadin-bom:18.0.6')
implementation("com.vaadin:vaadin-core")
implementation group: 'com.github.appreciated', name: 'vaadin-css-grid', version: '2.0.0'
// Logging
implementation group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.13.1'
implementation group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.13.1'
implementation group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: '2.13.1'
// Testing
testImplementation group: 'junit', name: 'junit', version: '4.12'
testImplementation group: 'org.skyscreamer', name: 'jsonassert', version: '1.5.0'
// Jetty
implementation group: 'org.eclipse.jetty', name: 'jetty-server', version: '9.4.31.v20200723'
implementation group: 'org.eclipse.jetty', name: 'jetty-webapp', version: '9.4.31.v20200723'
implementation group: 'org.eclipse.jetty.websocket', name: 'websocket-server', version: '9.4.31.v20200723'
implementation group: 'org.eclipse.jetty.websocket', name: 'javax-websocket-server-impl', version: '9.4.31.v20200723'
}
All these logs are really annoying (they slow down the start up really much) and I can't understand if they are dangerous or not.
Which is the right way to avoid duplications ?
Is there a technique to instruct AnnotationParser to scan only some dependencies?
I know this topic is present in different questions but I didn't find a solution for gradle or a common solution strategy.
Thanks,
Stefano
Having the same class name in multiple locations on your classpath is a bad idea.
This is the most common form of unstable operation on Java there is!
The classloaders in Java have no guarantee of load order behavior if the same class is present in multiple locations within the classloader.
In one run you might accidentally load the classes in the order you intend, and have it run properly, then at a future date you run the same program and the load order is different, now you are running with a different class version and you have unexpected behavior.
The only way to fix this is to clean up your classloader and ensure that you only have 1 version of the class you intend to use.
This is what Jetty is telling you.
As for this specific one, javax.websocket-api and javax.websocket-client-api, you want to exclude javax.websocket-client-api at the gradle level, as all of the websocket client classes are also present in the javax.websocket-api.
The javax.websocket-client-api jar is only intended for projects that only use the javax.websocket Client, without a javax.websocket Server.
Following the suggestion of joakim-erdfelt
I have modified my build.gradle and this prevent the problem:
implementation ('org.eclipse.jetty.websocket:javax-websocket-server-impl:9.4.31.v20200723') {
exclude module: 'javax.websocket-client-api'
}
The Gradle documentation is here: Excluding transitive dependencies
This is for sure a duplicate question, but I am not understanding what's happening.
I am trying to use the apicurio-registry library.
I added the following line in my build.gradle file :
implementation group: 'io.apicurio', name: 'apicurio-registry', version: '1.3.1.Final', ext: 'pom'
After building, I can see that I only retrieved the pom.xml, but not any of the jars specified in it (There is nothing except the pom.xml in the gradle caches).
I tried import hbase 2.1.0 of cloudera 6.3.3 at my gradle file like this:
compile ("org.apache.hbase:hbase-client:2.1.0-cdh6.3.3"){
exclude group: 'org.slf4j'
exclude group: 'org.jruby'
exclude group: 'jruby-complete'
exclude group: 'org.codehaus.jackson'
exclude group: 'org.codehaus.jettison'
}
When I refresh the gradle , it shows below error:
Could not resolve org.apache.hbase:hbase-client:2.1.0-cdh6.3.3.
I tried refreshing gradle dependencies , but no luck
Any help appreciated! Thanks in Advance!
When you are in doubt about dependencies like this, use a repository aggregator like mvnrepository and search for the module. You can find version 2.1.0-cdh6.3.3 of HBase Client here:
As you can see from the description, the artifact is located in the Cloudera Maven repository, so you will need to configure that in Gradle:
repositories {
maven {
url "https://repository.cloudera.com/artifactory/cloudera-repos/"
}
}
Also, don't use the compile configuration as it is deprecated. Use implementation or similar instead.
I am trying to integrate Spark application with spring boot but since spark core also has jetty server and servlet packages, they are conflicting with spring boot web starter servlet packages.
I already followed the post below to exclude starter-logging
https://www.linkedin.com/pulse/integrating-spark-spring-boot-neeraj-malhotra
I tried excluding servlet from spark core package, but it wouldn't start up or build using gradle build.
My goal is to run this app as a Rest API using spring boot web starter.
Thanks
I found the solution a long time ago, but couldn't post this.
I had to exclude a bunch of packages from spark-core that contain servlet packages
compile(group: 'org.apache.spark', name: 'spark-core_2.11', version: '1.6.0') {
exclude group: 'org.slf4j', module: 'slf4j-simple'
exclude group: 'org.eclipse.jetty', module: 'jetty-server'
exclude group: 'com.sun.jersey', module: 'jersey-server'
exclude group: 'com.sun.jersey', module: 'jetty-core'
exclude group: 'org.eclipse.jetty.orbit', module: 'javax.servlet'
//exclude group: 'org.eclipse.jetty', module: 'jetty-server'
}
I use gradle and latest.release everywhere to be promiscuous, however, recently my Spring boot application stopped running from command line. When I run gradle clean bootRun I get...
Exception in thread "main" java.lang.IllegalArgumentException: LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.slf4j.impl.Log4jLoggerFactory loaded from file:/.../.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.25/110cefe2df103412849d72ef7a67e4e91e4266b4/slf4j-log4j12-1.7.25.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.slf4j.impl.Log4jLoggerFactory
Per this post I tried...
configurations.all {
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-tomcat'
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
exclude group: 'org.springframework.boot', module: 'logback-classic'
exclude group: "org.slf4j"
}
and that silences the Logback error but spring won't start and fails with no error.
Caused by: org.gradle.process.internal.ExecException: Process 'command '/Library/Java/JavaVirtualMachines/jdk-10.0.1.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1
What am I missing?
I had this and fixed it by excluding spring-boot-starter-logging and logback-classic and adding spring log4j2 implementation in gradle.build
configurations {
all {
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
exclude group: 'ch.qos.logback', module: 'logback-classic'
}
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-log4j2:2.2.6.RELEASE'
Needed to remove.. compile 'org.apache.spark:spark-sql_2.11:latest.release'
Haven't figured out through why.
Spring Boot automatically configures the Log Back library because of the web starter. You should exclude the spring-boot-starter-logging to fix the issue.
To exclude the default LogBack configuration on the gradle project,
configurations {
all*.exclude module : 'spring-boot-starter-logging'
all*.exclude module : 'logback-classic'
}
You can also try module replacement to replace the LogBack to Log4j or Log4j2