use maven to compile ycsb - maven

I want to run ycsb to test hbase, and I referred to this blog:
http://cloudfront.blogspot.in/2013/02/how-to-benchmark-hbase-using-ycsb.html#.Uy-J2XiLe1F
But when I changed pox.xml in /ycsb/hbase, I try to build ycsb using maven, then I got this error:
[ERROR] Failed to execute goal on project hbase-binding: Could not resolve dependencies for project com.yahoo.ycsb:hbase-binding:jar:0.1.4: Failure to find org.apache.hadoop:hadoop-core:jar:2.3.0 in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
I've already successfully installed hadoop-2.3.0 and hbase-0.94.17.
Here is the pom.xml I use:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.yahoo.ycsb</groupId>
<artifactId>root</artifactId>
<version>0.1.4</version>
</parent>
<artifactId>hbase-binding</artifactId>
<name>HBase DB Binding</name>
<dependencies>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.94.17</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>${maven.assembly.version}</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>false</appendAssemblyId>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Any help would be appreciated, thanks!

The problem is sometimes maven can't get the jars for hbase & hadoop. You will have to download them manually and add it to classpath.
Source: Experimenting with YCSB extensively :)

Related

Error running stand-alone jar with ojdbc jar dependency

I have the following pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>do_can_proj</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.oracle.database.jdbc</groupId>
<artifactId>ojdbc11-production</artifactId>
<version>21.1.0.0</version>
<type>pom</type>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<archive>
<manifest>
<mainClass>com.do.can.Main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
Running,
$ mvn clean package assembly:single
generates a jar in the target folder.
When I run the jar file from the command line:
java -jar target/do_can_proj-1.0-SNAPSHOT-jar-with-dependencies.jar
I get the following error:
SQL State: 08001
No suitable driver found for jdbc:oracle:thin:#hostname:1521:SID
And this is how I have my project setup:
When running with the -jar argument, the classpath is set to the entry in the manifest in the jar file. By default this is empty. It can be argued that the assembly target should be improved to allow this.
It needs to be set to include all libraries used by your code for this to work and the libraries needs to be added from the Maven repository. How exactly to do this depends on your needs. See How can I create an executable JAR with dependencies using Maven? for suggestions.

How to resolve a conflict with a system scope dependency jar

I have a jar that my code depends on and it is located locally on my PC.
That jar contains the 'org.apache.commons.lang3' package but an older version or a version that contains the class 'StringUtils' but without all of its methods.
In my maven pom, I am using the 'org.apache.commons.lang3' dependency also but at runtime I get the error 'NoSuchMethodError' which means that during runtime it is using the commons.lang3 version of the jar and not the one from my dependency.
Is there a way to force it to use the dependency in my pom file?
Here is how my pom looks like:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test</groupId>
<artifactId>test</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<test.path>C:/Users/user/test</test.path>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>dep</groupId>
<artifactId>dep</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${test.path}/dep.jar</systemPath>
</dependency>
</dependencies>
<!-- build fat jar file with dependencies -->
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>dep-included</shadedClassifierName>
<outputDirectory>${test.path}/out</outputDirectory>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
CONCLUSION FROM COMMENTS:
It is not possible to remove such packages in a jar with Maven. I have to manualy edit and remove the packages from the jar file.
Thank you all for taking the time to answer!

Packaging with maven-assembly-plugin: JNI error, then "an exception has occured"

Edit: The same issue happens even when packaging without Maven via intelliJ, see here. JDK I built the project in and my environment variables are both Amazon Corretto 11. I'm not able to get a stacktrace because the program works perfectly through cmd. This issue only happens when trying to click to open the jar
I've spent all day trying to package my Maven project. I've been trying to use the maven-assembly-plugin. I believe I've followed this and this respectively, though it's possible I missed something. My entire pom.xml is here, as I'm guessing the answer will lie there.
The exact errors I get when trying to run the jars via clicking are: "Error: A JNI error has occurred, please check your installation and try again" followed by "A Java Exception has occurred". The code runs fine when launched through my IDE and definitely doesn't throw a compile time exception as it appears to be doing when running as a jar.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>groupId</groupId>
<artifactId>ScraperTest</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>Main</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase> <!-- bind to the packaging phase -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.13.1</version>
</dependency>
<dependency>
<groupId>org.xerial</groupId>
<artifactId>sqlite-jdbc</artifactId>
<version>3.32.3.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.11</version>
</dependency>
</dependencies>
</project>

What dependency scope for Derby JDBC drivers in intergration scope (cayenne)

I am trying to build my pom.xml so that I can automatically create my database schema when running 'mvn install'. I'm using the "maven-cayenne-plugin" to do this. This is plugin is being called (at the integration-test phase), as I can see the output. But the mojo fails with the exception: (I used the -e and -X flag to see this).
java.lang.ClassNotFoundException: org.apache.derby.jdbc.ClientDriver
(I get the same if I try and use the EmbeddedDriver and whether or not I include 'derbyclient' or simply 'derby' as my dependency).
Here's a pom.xml that should replicate the issue.
I'm using MVN 3 on Windows.
[ Apache Maven 3.0.4 (r1232337; 2012-01-17 08:44:56+0000) ]
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>myproject</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<build>
<plugins>
<plugin>
<groupId>org.apache.cayenne.plugins</groupId>
<artifactId>maven-cayenne-modeler-plugin</artifactId>
<version>3.2M1</version>
</plugin>
<plugin>
<groupId>org.apache.cayenne.plugins</groupId>
<artifactId>maven-cayenne-plugin</artifactId>
<version>3.2M1</version>
<executions>
<execution>
<id>cgen</id>
<configuration>
<superPkg>com.mycompany.model.generated</superPkg>
<map>${project.build.sourceDirectory}/../resources/datamap.map.xml</map>
<destDir>${project.build.sourceDirectory}</destDir>
</configuration>
<goals>
<goal>cgen</goal>
</goals>
</execution>
<execution>
<id>cdbgen</id>
<configuration>
<map>${project.build.sourceDirectory}/../resources/datamap.map.xml</map>
<driver>org.apache.derby.jdbc.ClientDriver</driver>
<url>jdbc:derby:memory:tracedb;create=true</url>
<username>test</username>
</configuration>
<goals>
<goal>cdbgen</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derbyclient</artifactId>
<version>10.10.1.1</version>
<scope>compile</scope>
</dependency>
</dependencies>
</project>
This also requires a valid cayenne "datamap.map.xml" file (in src/main/resources), here's one I made earlier:
<?xml version="1.0" encoding="utf-8"?>
<data-map xmlns="http://cayenne.apache.org/schema/3.0/modelMap"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://cayenne.apache.org/schema/3.0/modelMap http://cayenne.apache.org/schema/3.0/modelMap.xsd"
project-version="6">
<db-entity name="TEST">
<db-attribute name="ID" type="INTEGER" isPrimaryKey="true" isMandatory="true"/>
</db-entity>
</data-map>
EDIT:
Adding more information.
The derbyclient-10.10.1.1.jar does contain the class 'org.apache.derby.jdbc.ClientDriver' (just expanded the JAR from Netbeans).
The -X flag seems to show that the CLASSPATH is correctly referencing the JAR:
[DEBUG] (f) classpathElements = [<PROJECT-PATH>\mvn\target\classes, <HOME-DIR>\.m2\repository\org\apache\derby\derbyclient\10.10.1.1\derbyclient-10.10.1.1.jar]
SOLUTION:working pom.xml (see answer and my comment):
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>myproject</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<build>
<plugins>
<plugin>
<groupId>org.apache.cayenne.plugins</groupId>
<artifactId>maven-cayenne-modeler-plugin</artifactId>
<version>3.2M1</version>
</plugin>
<plugin>
<groupId>org.apache.cayenne.plugins</groupId>
<artifactId>maven-cayenne-plugin</artifactId>
<version>3.2M1</version>
<dependencies>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>10.10.1.1</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>cgen</id>
<configuration>
<superPkg>com.mycompany.model.generated</superPkg>
<map>${project.build.sourceDirectory}/../resources/datamap.map.xml</map>
<destDir>${project.build.sourceDirectory}</destDir>
</configuration>
<goals>
<goal>cgen</goal>
</goals>
</execution>
<execution>
<id>cdbgen</id>
<configuration>
<map>${project.build.sourceDirectory}/../resources/datamap.map.xml</map>
<driver>org.apache.derby.jdbc.EmbeddedDriver</driver>
<url>jdbc:derby:memory:tracedb;create=true</url>
<username>test</username>
</configuration>
<goals>
<goal>cdbgen</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derbyclient</artifactId>
<version>10.10.1.1</version>
<scope>compile</scope>
</dependency>
</dependencies>
</project>
To ensure that the Derby driver is available during plugin execution (vs during your code compilation), you need to add it as a dependency of the plugin itself:
<plugin>
<groupId>org.apache.cayenne.plugins</groupId>
<artifactId>maven-cayenne-plugin</artifactId>
<version>3.2M1</version>
<dependencies>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derbyclient</artifactId>
<version>10.10.1.1</version>
</dependency>
</dependencies>
....
</plugin>

run findbugs through maven eclipse

how to run findbugs through eclipse with maven project. I have configured in maven as:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.home.app</groupId>
<artifactId>home-app</artifactId>
<version>0.0.1</version>
<packaging>jar</packaging>
<name>home-app</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<effort>Max</effort>
<threshold>Low</threshold>
<xmlOutput>true</xmlOutput>
</configuration>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
i want to know what are the errors/warnings reported by findbugs.
You will find the report during the usual compile package phase created in the target directory. If you like to see a more readable output you have to use the findbugs goal in the reporting block instead of the build block.
You could also just use the findbugs eclipse plugin which will give you a nice eclipse window that points you directly to all of your bugs.

Resources