I'm trying to upgrade from Mongo 3.2 to Mongo 3.4. I updated my POM to use the latest Spring Data MongoDB (1.10.13) and while doing so, I noticed Spring Data MongoDB is still being compiled/depended on the old MongoDB Java driver (2.14.3). Since my project is dependent on the Spring Data MongoDB and MongoDB 3.4 driver, I ended up having 2 MongoDB drivers in my libs, which seems to have caused problems.
I ended up adding an exclusion in my POM. Is this the right way to go? How should I override the old MongoDB driver which originates from Spring Data MongoDB 1.10?
This is what my POM looks like:
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.4.3</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-mongodb</artifactId>
<version>1.10.13.RELEASE</version>
<exclusions>
<exclusion>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
</exclusion>
</exclusions>
</dependency>
I configured my Nexus to proxy maven.oracle.com and I am able to browse the index, so I know it worked. Oracle's documentation from this year (https://blogs.oracle.com/dev2dev/entry/oracle_maven_repository_instructions_for) gives this as the dependency for their driver:
<dependency>
<groupId>com.oracle.jdbc</groupId>
<artifactId>ojdbc7</artifactId>
<version>12.1.0.2</version>
</dependency>
But I cannot find it and Maven can't resolve it. I do find a jar ojdbc7 under a different GAV in their maven repo though:
<dependency>
<groupId>com.oracle.weblogic</groupId>
<artifactId>ojdbc7</artifactId>
<version>12.1.3-0-0</version>
</dependency>
Have I configured it wrong? Or is their doc wrong?
There is a question here connect from java to Hive but mine is different
My hive running on machine1 and I need to pass some queries using Java server running at machine2. As I understand Hive has a JDBC interface for the purpose of receiving remote queries. I took the code from here - HiveServer2 Clients
I installed the dependencies written in the article:
hive-jdbc*.jar
hive-service*.jar
libfb303-0.9.0.jar
libthrift-0.9.0.jar
log4j-1.2.16.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
commons-logging-1.0.4.jar
However I got java.lang.NoClassDefFoundError error at compile time
Full Error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:393)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:187)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at com.bidstalk.tools.RawLogsQuerySystem.HiveJdbcClient.main(HiveJdbcClient.java:25)
Another question at StackOverflow recommended to add Hadoop API dependencies in Maven - Hive Error
I don't understand why do I need hadoop API for a client to connect with Hive. Shouldn't JDBC driver be agnostic of the underlying query system? I just need to pass some SQL query?
Edit:
I am using Cloudera(5.3.1), I think I need to add CDH dependencies. Cloudera instance is running hadoop 2.5.0 and HiveServer2
But the servers are at machine 1. On machine the code should at least compile and I should have issues at runtime only!
In case if you didn't still solve this, I have given it a go.
And I needed the following dependencies for it to compile and run :
libthrift-0.9.0-cdh5-2.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
commons-logging-1.1.3.jar
hive-common.jar
slf4j-api-1.7.5.jar
hive-metastore.jar
hive-service.jar
hadoop-common.jar
hive-jdbc.jar
guava-11.0.2.jar
The hive documentation is probably written against a older version/distribution.
Your exception is due to the missing hadoop-common jar, which has the org.apache.hadoop.conf.Configuration.
Hope this helps.
Getting the same error when trying to use hive-jdbc 1.2.1 against hive 0.13.
Comparing to the long list in other answers. Now we use these two:
hive-jdbc-1.2.1-standalone.jar
hadoop-common-2.7.1.jar
Another side note: you might get 'Required field 'client_protocol' is unset!' when using the latest jdbc against an older Hive. If so, change the jdbc version to 1.1.0:
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.1.0</version>
<classifier>standalone</classifier>
</dependency>
Answering my own question!
With some hit and trial, I have added following dependencies on my pom file and since then I am able to run code on both CHD 5.3.1 and 5.2.1 cluster.
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>0.13.1-cdh5.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libfb303</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.5.0-mr1-cdh5.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.5.0-cdh5.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>0.13.1-cdh5.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.5.0-cdh5.3.1</version>
</dependency>
<dependency>
Please note that some of these dependencies might not be required
For others wondering around about what exactly is required to execute remotely a HIVE query using java...
Java code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
public class Runner
{
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
try {
// Register driver and create driver instance
Class.forName(driverName);
} catch (ClassNotFoundException ex) {
ex.printStackTrace();
}
// get connection
System.out.println("before trying to connect");
Connection con = DriverManager.getConnection("jdbc:hive2://[HOST IP]:10000/", "hive", "");
System.out.println("connected");
// create statement
Statement stmt = con.createStatement();
// execute statement
stmt.executeQuery("show tables");
con.close();
}
}
Together with the pom file with the only required dependencies..
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test-executor</groupId>
<artifactId>test-executor</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<hadoop.version>2.5.2</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
</project>
I have faced the same issue with CDH5.4.1 version. I updated my POM file with the below code and it worked for me.
My Hadoop Version is Hadoop 2.6.0-cdh5.4.1 and Hive version is Hive 1.1.0-cdh5.4.1
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>0.13.0</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>0.13.0</version>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libfb303</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.0</version>
</dependency>
I have resolved with this POM update.
Seems like you are all working with cloudera, I found that the repo in maven looks old because if you go to their site, you can download their jdbc. https://www.cloudera.com/downloads/connectors/hive/jdbc/2-5-20.html
The driver seems to support more functionality than the one in hive. I notice that that they have addBatch implemented. I just wish they had these libraries in maven. Maybe someone can find where to get them from using maven.
Softer version of cdh3 client interoperable with apache hadoop server 0.20.xx?
We have a java app that runs a series of pig scripts (it injects some variables in them, but generally it's just a driver/client for running them).
We need the macro features of pig 0.9.0 but cdh comes with pig 0.8.1.
Is the following a good idea?
We try working with:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2-cdh3u1</version>
</dependency>
with
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.9.0</version>
</dependency>
instead of
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.8.1-cdh3u1</version>
</dependency>
We have a standalone desktop client that connects to a JBoss server. For version 6 of JBoss the maven dependency used by the desktop client project was
<dependency>
<groupId>org.jboss.jbossas</groupId>
<artifactId>jboss-as-client</artifactId>
</dependency>
For JBoss 7.1.1 no such dependency exists. What is the correct maven dependency that should be used when developing a standalone desktop client?
If you directly connect to EJB you need the EJB client libs.In earlier versions of JBoss AS7 there were a bunch of individual dependencies required. Starting (AFAIK) from 7.1.1-Final a BOM (bill of materials) is available:
<dependencies>
<dependency>
<groupId>org.jboss.as</groupId>
<artifactId>jboss-as-ejb-client-bom</artifactId>
<version>7.1.1.Final</version>
<type>pom</type>
</dependency>
</dependencies>
You will find here detailed information on JNDI lookups and invoking methods.
I'm not too familiar with the JBoss AS 6 client, but for the JBoss AS 7 one you would need the following.
<dependencies>
<dependency>
<groupId>org.jboss.as</groupId>
<artifactId>jboss-as-controller-client</artifactId>
<version>${as.version}</version>
</dependency>
</dependencies>
You'll be using the org.jboss.as.controller.client.ModelControllerClient for standalone or org.jboss.as.controller.client.helpers.domain.DomainClient for domain mode.