I'm using small program to write data into Accumulo.
Program worked when added jars manually. But, when build with maven, with the same version's used from manual it throws:
java.lang.NoClassDefFoundError:
org/apache/accumulo/core/client/Instance.
How would I resolve it?
Your job is going to be run on all of the nodes in your MR network. You'll need the appropriate jars on all of the nodes in order for it to work.
Another approach, as you have noticed, is to just include everything into one uber jar, which contains everything that you need. That way when your job gets shipped to each node, you'll have everything you need. One way to accomplish this with maven is through the use of plugins:
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2.1</version>
<configuration>
<descriptors>
<descriptor>src/main/assembly/hadoop-job.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Inside of your hadoop-job.xml you might have
<assembly>
<id>job</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>false</unpack>
<scope>runtime</scope>
<outputDirectory>lib</outputDirectory>
<excludes>
<exclude>${groupId}:${artifactId}</exclude>
</excludes>
</dependencySet>
<dependencySet>
<outputDirectory></outputDirectory>
<unpack>true</unpack>
<includes>
<include>${groupId}:${artifactId}</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>
More information about this plugin may be found at http://maven.apache.org/plugins/maven-assembly-plugin/
You have to download the accumulo-core.jar and add it to your classpath.
Related
My goal is to create a jar with specific dependencies from my dependency list in the pom. I'm using maven-dependency-plugin like so:
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<goals>
<goal>unpack-dependencies</goal>
</goals>
<phase>prepare-package</phase>
<configuration>
<includeScope>runtime</includeScope>
<excludes>META-INF/*.SF,META-INF/*.DSA,META-INF/*.RSA</excludes>
<outputDirectory>${project.build.directory}/uber-deps/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.some.blaClass</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
and an assembly.xml file holding:
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
<id>plugin</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<useProjectArtifact>true</useProjectArtifact>
<unpack>true</unpack>
<scope>runtime</scope>
<includes>
<include>
${project.build.directory}/uber-deps/
</include>
</includes>
<excludes>
<exclude>*:sources</exclude>
</excludes>
</dependencySet>
</dependencySets>
</assembly>
After mvn clean install all relevant dependencies appear in target/uber-deps as I would expect. My problem is with the next plugin under <plugins> - maven-assembly-plugin. Seems to me as if it doesn't take uber-deps in.
I know this only by trying to unpack the jar using jar xf to see if the deps in uber-deps were packed in the jar created after mvn clean install.
What should be changed?
1)
The jar you are building as part of the assembly-plugin will be called (by default) ./target/<artifactId>-plugin.jar
Note that the plugin part is what you've put under id in your assembly xml file.
2)
Since you already unpack the dependencies to a folder, you should use fileSets rather then dependencySets:
<fileSets>
<fileSets>
<fileSet>
<outputDirectory>/</outputDirectory>
<includes>
<include>
${project.build.directory}/uber-deps/
</include>
</includes>
</fileSet>
</fileSets>
</fileSets>
3)
BTW if you want the outputs of your own project in that jar you should add another fileSet:
<fileSet>
<outputDirectory>/</outputDirectory>
<includes>
<include>
${project.build.outputDirectory}
</include>
</includes>
</fileSet>
4) Also just noted that your assembly plugin definition is not stating the location of your assembly xml file and that you try to define mainClass using shade-plugin configuration. This is how it should look in assembly plugin (assuming that your assembly file is located under src/assembly/plugin.xml:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<configuration>
<descriptors>
<descriptor>src/assembly/plugin.xml</descriptor>
</descriptors>
<archive>
<manifest>
<mainClass>com.some.blaClass</mainClass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>
I am trying to create a runnable Jar file. I have to create a Jar of only one single java class (MyClass which is under the folder structure src/main/java/org/miso/mcs/util/MyClass) and exclude others. I am able to create to JAR but it has everything.
I am using mvn clean package assembly:single command to create Jar.
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>org.miso.mcs.util.MyClass</mainClass>
</manifest>
</archive>
<resources>
<resource>
<classesDirectory>src/main/java/org/miso/mcs/util</classesDirectory>
<includes>
<include>/*MyClass*</include>
</includes>
<excludes>
<exclude>/*</exclude>
</excludes>
</resource>
</resources>
</configuration>
</plugin>
You won't be able to do that by using the predefined jar-with-dependencies descriptor. However, we can make our own assembly descriptor for that:
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>jar-with-dependencies</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<useProjectArtifact>false</useProjectArtifact>
<unpack>true</unpack>
</dependencySet>
<dependencySet>
<outputDirectory>/</outputDirectory>
<useProjectArtifact>false</useProjectArtifact>
<useProjectAttachments>true</useProjectAttachments>
<includes>
<include>${project.groupId}:${project.artifactId}:jar:classes:${project.version}</include>
</includes>
<unpack>true</unpack>
<unpackOptions>
<excludes>
<exclude>%regex[org\/miso\/(?!mcs\/util\/MyClass).*]</exclude>
</excludes>
</unpackOptions>
</dependencySet>
</dependencySets>
</assembly>
unpackOptions allows to control the content the unpacked dependencies in the JAR. In this case, we are excluding everything under the org.miso package (which I assume are the classes of your project) and we're only including the class you're interested in (org.miso.mcs.util.MyClass). This will result in a runnable JAR with only the specified class of your project.
Since your project is a WAR, it is a little more complicated. The first dependency set declares all the dependencies of the project and unpacks them. The second only uses the projects attachments and includes only the classes attachment built by the maven-war-plugin.
If you save that snippet into a file called assembly.xml located under src/assembly, your POM will then look like:
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>2.6</version>
<configuration>
<attachClasses>true</attachClasses>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/assembly/assembly.xml</descriptor>
</descriptors>
<archive>
<manifest>
<mainClass>org.miso.mcs.util.MyClass</mainClass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>
With such a configuration, you just need to invoke Maven with mvn clean install and it will correctly generate the JAR. Then, you can launch that JAR with java -jar name-of-jar.jar.
I have to create multiple packages for a single application built by Maven. At first I use the -Denv to generate different packages by Maven arguments and I use the resource to include/exclude the resources:
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.properties</include>
<include>**/${env}.ini></include>
</includes>
<filtering>true</filtering>
</resource>
</resources>
Then I can use
mvn clean package -Denv=dev
mvn clean package -Denv=prod1
mvn clean package -Denv=prod2
mvn clean package -Denv=prod3
.....
However it seems that different packages have to be packaged one by one, which means Maven will spent too much time to build all the packages since we have almost 22 different environments to deploy the application.
Furthermore, each mvn .. package almost generate the same result like classes,resources except some specified resources like {$env}.ini, I think it is a waste to run the compile again and again.
It would be better if we compile all the sources, and resources to the target directory, then generate different packages accordingly, so I thought the maven-assembly-plugin, generally we will use it like this:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptors>
<descriptor>/src/main/assembly/env.xml</descriptor>
<descriptor>/src/main/assembly/prod1.xml</descriptor>
<descriptor>/src/main/assembly/prod2.xml</descriptor>
......
</descriptors>
</configuration>
</plugin>
Then we have to create 22 descriptors, however these descriptors are too similar with each other, which means there are too many repeated configuration, and it will be difficult once I want to make some general change to all the descriptors.
So I wonder if there is a good alternative to solve this kind of requirements?
You can use the iterator-maven-plugin to solve this problem like this:
<plugin>
<groupId>com.soebes.maven.plugins</groupId>
<artifactId>iterator-maven-plugin</artifactId>
<version>0.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>iterator</goal>
</goals>
<configuration>
<items>
<item>test</item>
<item>prod</item>
<item>dev</item>
</items>
<pluginExecutors>
<pluginExecutor>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
</plugin>
<goal>single</goal>
<configuration>
<descriptors>
<descriptor>${project.basedir}/#item#.xml</descriptor>
</descriptors>
</configuration>
</pluginExecutor>
</pluginExecutors>
</configuration>
</execution>
</executions>
</plugin>
If you use the descriptor like this: ${project.basedir}/#item#.xml the name will be replaced by every single iteration step so you can have several descriptors...but in your case i assume that you have very similar descriptors so you have the same descriptors without a placeholder which contains only slight differences which can be solve like this in the descriptor itself:
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
<id>${item}</id>
<formats>
<format>war</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>true</unpack>
<useProjectArtifact>true</useProjectArtifact>
</dependencySet>
</dependencySets>
<fileSets>
<fileSet>
<outputDirectory>WEB-INF</outputDirectory>
<directory>${basedir}/src/main/environment/${item}/</directory>
<includes>
<include>**</include>
</includes>
</fileSet>
</fileSets>
</assembly>
as a result you need only a single descriptor. If you have any issues please create issue requests.
I am having trouble building a jar using the assembly plugin. I have a bunch of system jar files that I want bundled alongside class files of the project.
I was looking to do something like:
final
- lib
- system.jar
- system1.jar
- com
- servlet
-- etc
I have tried to use the assembly plugin but it generates 2 jars. One with just the lib with my jars and another with the class files.
I have read through the assembly docs but I must be missing something.
Here is the build element I have in my pom.xml
<dependency>
<groupId>jackson-mapper-asl</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.4</version>
<scope>system</scope>
<systemPath>${project.basedir}/lib/jackson-mapper-asl-1.9.4-sc1.jar</systemPath>
</dependency>
//... etc
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>final-jar</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>assembly.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
This is my assembly file (assembly.xml)
<assembly>
<id>distribution</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<useProjectArtifact>false</useProjectArtifact>
<useTransitiveDependencies>true</useTransitiveDependencies>
<unpack>false</unpack>
<scope>system</scope>
<outputDirectory>lib</outputDirectory>
<includes />
</dependencySet>
</dependencySets>
</assembly>
Any idea how I can merge these jars together?
You would usually have a much simpler layout, such as:
final
- lib
-system.jar
-system1.jar
-project.jar
I.e. the project itself is a jar too, and included in the lib. In this case just add <useProjectArtifact>true/>useProjectArtifact> to the dependencySet.
If you really, really want to have classes copied in directly, try adding a fileset to the assembly, copying from target/classes into your root location
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>/</outputDirectory>
</fileSet>
</fileSets>
I have a java project XXX
src/main/java
src/main/config
src/main/scripts
I want output structure like
C:/target/XXX.jar
C:/target/scripts
I tried to use resource plugin but it is packing everything inside jar.
<build>
<directory>target</directory>
<outputDirectory>target/classes</outputDirectory>
<finalName>${artifactId}-${version}</finalName>
<sourceDirectory>src/main/java</sourceDirectory>
<scriptSourceDirectory>src/main/scripts</scriptSourceDirectory>
<resources>
<resource>
<directory>src/main/config</directory>
</resource>
<resource>
<directory>src/main/scripts</directory>
<targetPath>/scripts</targetPath>
</resource>
</resources>
</build>
The maven-assembly-plugin is great for this sort of thing. You can create a descriptor something like this (in src/main/assemble/scripts.xml):
<assembly>
<id>scripts</id>
<formats>
<format>dir</format>
</formats>
<fileSets>
<fileSet>
<outputDirectory>/scripts</outputDirectory>
<includes>
<include>*.*</include>
</includes>
</fileSet>
</fileSets>
</assembly>
Wire this into your build like so:
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.3</version>
<configuration>
<descriptors>
<descriptor>src/main/assemble/scripts.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>copy-scripts</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
This will bind the execution of the assembly to the package phase of the build.
This method will give you the flexibility to change how you package up the scripts in the future, or copy more resources in the build.