Scala version error with Spark 2 & ElasticSearch 5.4.2 - hadoop

I'm using Spark 2.2 (build with Scala 2.11.8) to index my data into ElasticSearch 5.4.2.
ElasticSearch :
My project spark use this pom.xml :
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-hadoop</artifactId>
<version>5.4.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.11</artifactId>
<version>5.4.2</version>
</dependency>
When I run my job, I get this exception :
Caused by: java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
at org.elasticsearch.spark.serialization.ReflectionUtils$.org$elasticsearch$spark$serialization$ReflectionUtils$$checkCaseClass(ReflectionUtils.scala:42)
at org.elasticsearch.spark.serialization.ReflectionUtils$$anonfun$checkCaseClassCache$1.apply(ReflectionUtils.scala:84)
at org.elasticsearch.spark.serialization.ReflectionUtils$$anonfun$checkCaseClassCache$1.apply(ReflectionUtils.scala:83)
I know my problem is Scala version (build/run) ...
Thanks for your help
EDIT BUILD POM :
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.3.1</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<argLine>-J-Xms128m</argLine>
<argLine>-J-Xmx512m</argLine>
<argLine>-J--XX:MaxPermSize=300m</argLine>
<argLine>-Djava.net.preferIPv4Stack=true</argLine>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>UBER</shadedClassifierName>
<artifactSet>
<includes>
<include>com.databricks:spark-csv_${scala.compact.version}</include>
<include>org.apache.commons:commons-csv</include>
<include>org.elasticsearch:elasticsearch-hadoop</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Related

Kotlin Querydsl: cannot find javax.annotation.proceesing.Generated

I am following this example https://felixzett.com/articles/minimal-maven-kotlin-querydsl-example/ to implement querydsl in a Springboot Kotlin Maven project. Using the pom mentioned, I managed to generate the q classes after running mvn compile and can run mvn clean install successfully as well, but when I try to run the project locally, I get error in all the q classes.
java: cannot find symbol
symbol: class Generated
location: package javax.annotation.processing
I tried adding javax.annotation-api to the <annotationProcessPaths> element but it didn't work.
<annotationProcessorPath>
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
<version>1.3.2</version>
</annotationProcessorPath>
I also tried adding it as a dependency, within that <plugin> element or in the main <dependencies> element, but these didn't work as well.
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
<version>1.3.2</version>
</dependency>
May I know if anyone has any idea how to solve this?
Update:
I have both kotlin-maven-plugin and maven-compiler-plugin in my pom. I realise if I comment out the maven-compiler-plugin then the error is gone. But I am not sure if other parts of the app will be affected if it doesn't have this plugin.
Does anyone know if it is ok to just have the kotlin-maven-plugin, or how I should modify maven-compiler-plugin for the 2 plugins to work together?
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>${kotlin.version}</version>
<executions>
<execution>
<id>compile</id>
<phase>process-sources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>kapt</id>
<phase>generate-sources</phase>
<goals>
<goal>kapt</goal>
</goals>
<configuration>
<sourceDirs>
<sourceDir>src/main/kotlin</sourceDir>
</sourceDirs>
<annotationProcessorPaths>
<annotationProcessorPath>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-apt</artifactId>
<version>${querydsl.version}</version>
<classifier>jpa</classifier>
</annotationProcessorPath>
</annotationProcessorPaths>
</configuration>
</execution>
<execution>
<id>test-compile</id>
<phase>test-compile</phase>
<goals>
<goal>test-compile</goal>
</goals>
<configuration>
<sourceDirs>
<sourceDir>src/test/kotlin</sourceDir>
<sourceDir>target/generated-sources/kapt/test</sourceDir>
</sourceDirs>
</configuration>
</execution>
</executions>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
<plugin>jpa</plugin>
<plugin>all-open</plugin>
</compilerPlugins>
<pluginOptions>
<option>all-open:annotation=javax.persistence.Entity</option>
<option>all-open:annotation=javax.persistence.MappedSuperclass</option>
<option>all-open:annotation=javax.persistence.Embeddable</option>
</pluginOptions>
<jvmTarget>1.8</jvmTarget>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-noarg</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
<executions>
<execution>
<id>default-compile</id>
<phase>none</phase>
</execution>
<execution>
<id>default-testCompile</id>
<phase>none</phase>
</execution>
<execution>
<id>java-compile</id>
<phase>process-sources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>java-test-compile</id>
<phase>test-compile</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
It seems like it is due to the Java version in the maven-compiler-plugin. After I changed it from 1.8 to 11, I do not get the error even if I have both kotlin-maven-plugin and maven-compiler-plugin.
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
Also change the java version to 11 in the <properties> element.
<properties>
<java.version>11</java.version>
</properties>

Can't generate QueryDSL's QClass when upgrading to 5.0.0 (com.querydsl.sql.Configuration is not registered)

I upgraded my QueryDSL's version to 5.0.0 (for the JTS upgrade) but now, the plugin throw an error when regenerating QClass using mvn clean install : com.querydsl.sql.Configuration is not registered.
Here is my configuration :
<plugin>
<groupId>com.mysema.maven</groupId>
<artifactId>apt-maven-plugin</artifactId>
<version>1.1.3</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>process</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/generated-sources</outputDirectory>
<processor>com.querydsl.apt.jpa.JPAAnnotationProcessor</processor>
<options>
</options>
</configuration>
</execution>
</executions>
</plugin>
Use the dependency below.
<dependency>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-apt</artifactId>
<version>${querydsl.version}</version>
<classifier>jakarta</classifier>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-jpa</artifactId>
<classifier>jakarta</classifier>
<version>${querydsl.version}</version>
</dependency>
You're encouraged to use:
<dependency>
<groupId>com.querydsl</groupId>
<artifactId>querydsl-apt</artifactId>
<version>5.0.0</version>
<scope>provided</scope>
<classifier>jpa</classifier>
</dependency>
Instead of the apt-maven-plugin for Querydsl 4.0.0 and higher.
Hello I quite have the same issue. In order to make it work I specified with the apt-maven-plugin only the folders where my entities were like here :
<plugin>
<groupId>com.mysema.maven</groupId>
<artifactId>apt-maven-plugin</artifactId>
<version>1.1.3</version>
<executions>
<execution>
<goals>
<goal>process</goal>
</goals>
<configuration>
<includes>
<include>common.persistence.entity.**</include>
<include>security.persistence.entity.**</include>
</includes>
</configuration>
</execution>
</executions>
<configuration>
<outputDirectory>target/generated-sources/annotations</outputDirectory>
<processors>
<processor>com.querydsl.apt.jpa.JPAAnnotationProcessor</processor>
</processors>
</configuration>
Don't know if this would help you but. You should giv it a try !

maven gmavenplus-plugin problems with groovy syntax

I am trying to customize maven build process using gmavenplus-plugin.
To be precise, I have a workig script in gmaven-plugin and I am trying to re-implement it in gmavenplus-plugin(which is advertised as a rewrite of GMaven)
My running gmaven code
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<phase>initialize</phase>
<goals>
<goal>execute</goal>
</goals>
<configuration>
<source>
println(project.version)
println([1, 2, 3])
</source>
</configuration>
</execution>
</executions>
</plugin>
My attempt to re-write it in gmavenplus:
<plugin>
<groupId>org.codehaus.gmavenplus</groupId>
<artifactId>gmavenplus-plugin</artifactId>
<version>1.7.0</version>
<executions>
<execution>
<phase>initialize</phase>
<goals>
<goal>execute</goal>
</goals>
</execution>
</executions>
<configuration>
<scripts>
println(project.version)
println([1, 2, 3])
</scripts>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.4.1</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</plugin>
Which fails with
[ERROR] Script1.groovy: 2: unexpected token: 1 # line 2, column 34.
[ERROR] println([1
[ERROR] ^
[ERROR]
[ERROR] 1 error
Any groovy syntax I try to use fails.
Update
CDATA does not help.
<scripts>
<![CDATA[
println(project.version)
println([1, 2, 3])
]]>
according to examples https://github.com/groovy/GMavenPlus/wiki/Examples
there should be <script> inside <scripts>:
use mvn gplus:execute with the following example pom.xml
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>test-gmavenplus</groupId>
<artifactId>test-gmavenplus</artifactId>
<packaging>pom</packaging>
<version>1.1.0-SNAPSHOT</version>
<name>test gmavenplus</name>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.gmavenplus</groupId>
<artifactId>gmavenplus-plugin</artifactId>
<version>1.7.0</version>
<executions>
<execution>
<goals>
<goal>execute</goal>
</goals>
</execution>
</executions>
<configuration>
<scripts>
<script><![CDATA[
println "hello `${project.name}`"
]]></script>
</scripts>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.5.7</version>
<type>pom</type>
<scope>runtime</scope>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>

Can't build working Java app with gRPC, Protobuf and BoringSSL to JAR using Maven

I have problem with building my Java app to jar file using Maven.
Application is using gRPC and Protobuf.
When I start my app in IntelliJ everything work just fine, problem is when I want to build jar with Maven... I don't have much experience with creating pom files.
I tried to find some solution but nothing works and I ended up with pom.xml as below:
[...]
<properties>
<grpc.version>1.17.1</grpc.version>
<protoc.version>3.5.1-1</protoc.version>
<netty.tcnative.version>2.0.13.Final</netty.tcnative.version>
[...]
</properties>
<dependencies>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty.tcnative.version}</version>
<scope>runtime</scope>
</dependency>
[...]
</dependencies>
<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.5.0.Final</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.5.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}
</pluginArtifact>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.4.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
<executions>
<execution>
<id>enforce</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireUpperBoundDeps/>
</rules>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>pl.test.Main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
And this is the exception when I try to start gRPC server with SSL context:
Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: failed to load the required native library
at io.netty.handler.ssl.OpenSsl.ensureAvailability(OpenSsl.java:346)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:202)
at io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:43)
at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:347)
at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:335)
at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:422)
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:447)
at pl.test.grpc.GrpcServer.start(GrpcServer.java:80)
at pl.test.app.Main.lambda$new$0(Main.java:80)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:194)
at pl.test.grpc.GrpcServer.getSslContextBuilder(GrpcServer.java:72)
... 3 more
I'm building it using command:
mvn clean compile assembly:single
Can someone help with creating working pom file? The result doesn't have to be single jar file, it might be multiple jars.
I found solution, maybe it will help someone deal with the same problem.
We have to add dependency io.netty.netty-handler and set compatible versions of io.grpc.grpc-netty, io.netty.netty-tcnative-boringssl-static and io.netty.netty-handler as it is described in table over here https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty.
Here's my current pom.xml
[...]
<properties>
<grpc.version>1.17.1</grpc.version>
<protoc.version>3.5.1-1</protoc.version>
<netty.tcnative.version>2.0.17.Final</netty.tcnative.version>
<netty.handler.version>4.1.30.Final</netty.handler.version>
[...]
</properties>
<dependencies>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty</artifactId>
<version>${grpc.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>${netty.handler.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty.tcnative.version}</version>
<scope>runtime</scope>
</dependency>
[...]
</dependencies>
<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.5.0.Final</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.5.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}
</pluginArtifact>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.4.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
<executions>
<execution>
<id>enforce</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireUpperBoundDeps/>
</rules>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>pl.test.Main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
And now I can build jar using command:
mvn clean compile assembly:assembly

gmaven-plugin works for groovy 1.7.5 but not for 2.1.0

I have working maven 2 setup which compiles jUnit tests written in groovy. Both java and groovy tests are located at /src/test/java
See a snapshot of the pom.xml
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>1.3</version>
<executions>
<execution>
<id>testCompile</id>
<goals>
<goal>testCompile</goal>
</goals>
<configuration>
<sources>
<fileset>
<directory>${pom.basedir}/src/test/java</directory>
<includes>
<include>**/*.groovy</include>
</includes>
</fileset>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>1.7.5</version>
<scope>test</scope>
</dependency>
When I upgrade to plugin version 1.5 and groovy 2.1.0, */.groovy files are ignored. Has anybody met up with this problem?
I found this page https://confluence.atlassian.com/display/CLOVER/Compiling+Groovy+with+GMaven+plugin
Note that you must put your Groovy Classes and Tests under src/main/groovy and src/test/groovy respectively.
Following configuration based on that page seems to work:
<!-- Groovy and Maven https://confluence.atlassian.com/display/CLOVER/Compiling+Groovy+with+GMaven+plugin -->
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>${gmaven.version}</version>
<configuration>
<providerSelection>2.0</providerSelection>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.gmaven.runtime</groupId>
<artifactId>gmaven-runtime-2.0</artifactId>
<version>${gmaven.version}</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>${groovy.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<goals>
<goal>generateStubs</goal>
<goal>compile</goal>
<goal>generateTestStubs</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
And in dependencies of course
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>${groovy.version}</version>
</dependency>
And in properties
<properties>
<gmaven.version>1.5</gmaven.version>
<groovy.version>2.1.8</groovy.version>
</properties>
Ok, this configuration works for maven 2.
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>1.4</version>
<configuration>
<providerSelection>2.0</providerSelection>
<sourceEncoding>UTF-8</sourceEncoding>
</configuration>
<executions>
<execution>
<goals>
<goal>testCompile</goal>
</goals>
<configuration>
<sources>
<fileset>
<directory>${pom.basedir}/src/test/java</directory>
<includes>
<include>**/*.groovy</include>
</includes>
</fileset>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>2.0.0</version>
<scope>test</scope>
</dependency>
I experience the same problem, but downgrading to gmaven 1.4 solves the problem (using groovy-all 2.3.2)
First, each GMaven provider compiles against a particular version of Groovy, so there can be issues if Groovy breaks something with a point release. Second, GMaven is no longer maintained (that's why you don't see any providers for newer Groovy versions). I recommend switching to GMavenPlus or the Groovy-Eclipse compiler plugin for Maven.

Resources