java action to access kerberos Hive(SSL enabled) - hadoop

I have been trying to access Hive(Kerberos and SSL enabled) server from java action.
I need to perform some Hive actions like accessing Hive database and tables, working with hive partitions etc.
public static void main(String[] args) throws Exception, IOException {
Configuration conf=new Configuration();
UserGroupInformation.setConfiguration(conf);
conf.set("hadoop.security.authentication", "kerberos");
conf.set("hadoop.security.authorization", "true");
System.setProperty("java.security.auth.login.config","C:\\krb\\jaas.conf");
System.setProperty("sun.security.jgss.debug","true");
System.setProperty("javax.security.auth.useSubjectCredsOnly","false");
String principalConfig = System.getProperty("kerberosPrincipal", "hive/cdlk-tars-control#ABC.COM");
String keytab = System.getProperty("kerberosKeytab", "C:\\krb\\hive.keytab");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(principalConfig, keytab);
final UserGroupInformation ugi = UserGroupInformation.getLoginUser();
System.out.println("Login as: " + ugi.getUserName());
HiveConf hiveConf=new HiveConf();
hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, args[0]);
Hive hive = Hive.get(hiveConf);
hive.getConf().setIntVar(HiveConf.ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT, METASTORE_CLIENT_TIMEOUT);
List<String> database=hive.getAllDatabases();
//database.forEach(x->System.out.println(x));
}
Able to login, but unable to fetch Hive with metastore.
Login as: hive/cdlk-tars-control#ABC.COM
Exception in thread "main" org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:242)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:394)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:338)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:318)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:294)
at com.cdlk.util.hive.HiveTest.main(HiveTest.java:36)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4108)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237)
... 5 more
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3727)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3715)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2628)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
at com.sun.proxy.$Proxy10.getAllFunctions(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2562)
at com.sun.proxy.$Proxy10.getAllFunctions(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4105)
... 7 more
jass configuration
com.sun.security.jgss.initiate {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
principal="hive/cdlk-tars-control#ABC.COM"
doNotPrompt=true
keyTab="C:\krb\hive.keytab"
debug=false;
};
Please suggest the way to access Hive from java

Related

Unable to connect spark standlone application with kerberized hadoop

I am using Spark standalone 1.6.x version to connect kerberos enabled hadoop 2.7.x
JavaDStream<String> status = stream.map(new Function<String, String>() {
public String call(String arg0) throws Exception {
Configuration conf = new Configuration();
FileSystem fs = null;
conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.namenode.kerberos.principal", "hdfs/_HOST#REALM");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.setLoginUser(UserGroupInformation.loginUserFromKeytabAndReturnUGI("abc","~/abc.ketyab"));
System.out.println("Logged in successfully.");
fs = FileSystem.get(new URI(activeNamenodeURI), conf);
FileStatus[] s = fs.listStatus(new Path("/"));
for (FileStatus status : s) {
System.out.println(status.getPath().toString());
}
return "success";
}
});
But getting below exception
User : abc#REALM (auth:KERBEROS) Caused by: java.io.IOException:
Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
"hostname1/0.0.0.0"; destination host is: "hostname2":8020; at
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at
org.apache.hadoop.ipc.Client.call(Client.java:1472) at
org.apache.hadoop.ipc.Client.call(Client.java:1399) at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy44.create(Unknown Source) at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy45.create(Unknown Source) at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1725)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1668) at
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1593) at
org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397)
at
org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) at
com.abc.HDFSFileWriter.createOutputFile(HDFSFileWriter.java:354) ...
21 more Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS] at
org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at
org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521) at
org.apache.hadoop.ipc.Client.call(Client.java:1438) ... 43 more Caused
by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS] at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at
org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722) at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415)

How can i stop the connection from getting closed while reading data from Oracle DB using Spring Batch?

I have a Spring Batch with Spring boot Application in which I am reading data from external Oracle DB and Writing it to SQL Server in production environment. I am validating the connection before reading the data but even then getting Closed Connection issue.Is anyone aware of this issue ? Below is the Datasource config code and error logs.
#Slf4j
#Configuration
public class DataSourceReaderConfig {
private static final Logger LOGGER = LoggerFactory.getLogger(DataSourceReaderConfig.class);
#Autowired
private Environment env;
#SuppressWarnings("deprecation")
#ConfigurationProperties(prefix = "oracle.datasource")
#Bean(name="dataSourceReader")
#Primary
public DataSource dataSourceReader() {
LOGGER.info("Inside dataSourceReader Method");
DataSource datasource = new DataSource();
OracleDataSource ocpds = null;
try {
ocpds = new OracleDataSource();
} catch (SQLException e1) {
LOGGER.error("Error getting Oracle Datasource Connection = " + e1.getMessage());
}
ocpds.setURL(env.getProperty("oracle.datasource.url"));
ocpds.setUser(env.getProperty("oracle.datasource.username"));
ocpds.setPassword(env.getProperty("oracle.datasource.password"));
try {
ocpds.setImplicitCachingEnabled(true);
} catch (SQLException e2) {
LOGGER.error("Error Setting OracleDataSource ImplicitCachingEnabled = " + e2.getMessage());
}
try {
ocpds.setConnectionCachingEnabled(true);
} catch (SQLException e1) {
LOGGER.error("Error Setting OracleDataSource ConnectionCachingEnabled = " + e1.getMessage());
}
Properties cacheProps = new Properties();
cacheProps.setProperty("ValidateConnection","true");
try {
ocpds.setConnectionCacheProperties(cacheProps);
} catch (SQLException e) {
LOGGER.error("Error Setting OracleDataSource Connection Cache Properites = " + e.getMessage());
}
datasource.setDataSource(ocpds);
return datasource;
}
}
[0m[0m09:00:00,000 INFO [com.elm.salamah.scheduler.batch.configuration.CityBatchConfiguration] (pool-3130-thread-1) Job2 importCitiesJob Started at : Thu Dec 07 09:00:00 AST 2017
[0m[31m09:00:00,066 ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] (pool-3130-thread-1) Unexpected error occurred in scheduled task.: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLRecoverableException: Closed Connection
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy3242.getLastJobExecution(Unknown Source)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy3241.run(Unknown Source)
at com.elm.salamah.scheduler.batch.configuration.CityBatchConfiguration.performSecondJob(CityBatchConfiguration.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLRecoverableException: Closed Connection
at oracle.jdbc.driver.PhysicalConnection.needLine(PhysicalConnection.java:5389)
at oracle.jdbc.driver.OracleStatement.closeOrCache(OracleStatement.java:1578)
at oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:1563)
at oracle.jdbc.driver.OracleStatementWrapper.close(OracleStatementWrapper.java:94)
at oracle.jdbc.driver.PhysicalConnection.setTransactionIsolation(PhysicalConnection.java:4620)
at sun.reflect.GeneratedMethodAccessor276.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:79)
at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)
at com.sun.proxy.$Proxy3237.setTransactionIsolation(Unknown Source)
at org.springframework.jdbc.datasource.DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:193)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:256)
... 34 more

Impala Kerberos authentication issue with long running program

I developed a Java program that runs on a Hadoop Edge Node (Cloudera CDH 5.9.1). It runs "for ever" as it is a (micro)service replying to requests. It receives requests from a webapp running somewhere else and extract data from Impala using Impala JDBC driver and send back data.
It runs well during roughly 10 hours, then it becomes unable to get Impala JDBC connections. You will find stacktrace and the end of this question. In short, it throws javax.security.auth.login.LoginException: Unable to obtain Principal Name for authentication.
My understanding is that Impala JDBC driver becomes enable to authenticate against Kerberos because the Kerberos ticket expired. So to be sure that the user with who the program is launched has a Kerberos ticket, I added a piece of code before the JDBC connection creation. I read those 2 Stackoverflow threads :
Should I call ugi.checkTGTAndReloginFromKeytab() before every action on hadoop?
HBase Kerberos connection renewal strategy
They explained very well the issue and the solution. Here is my attempt to implement that (I don't understand why it fails for Impala) :
public class KerberosRelogger {
private final static Logger logger = LogManager.getLogger();
public static String userName;
public static String user;
public static String keytab;
public static Configuration conf;
private static UserGroupInformation ugi;
static {
try {
init();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public static void init() throws IOException {
userName = System.getProperty("user.name");
user = userName + "#MY_REALM";
keytab = System.getProperty("user.home") + "/" + userName + ".keytab";
conf = new Configuration();
conf.addResource(new Path("file:///etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("file:///etc/hadoop/conf/hdfs-site.xml"));
UserGroupInformation.setConfiguration(conf);
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(user, keytab);
logger.debug("userName : {}", userName);
logger.debug("user : {}", user);
logger.debug("keytab : {}", keytab);
}
public static void testHadoopConnexion() throws IOException {
ugi.checkTGTAndReloginFromKeytab();
FileSystem fs = FileSystem.get(conf);
FileStatus[] statuses = fs.listStatus(new Path("/"));
List<String> dirNames = new ArrayList<>();
for (FileStatus status : statuses) {
dirNames.add(status.getPath().getName());
}
logger.debug("HDFS root dirs : " + dirNames);
}
private static Connection createImpalaConnection() throws IOException {
ugi.checkTGTAndReloginFromKeytab();
String IMPALAD_HOST = "my_node";
String jdbcUrl = "jdbc:impala://" + IMPALAD_HOST + ":21050;AuthMech=1;KrbRealm=MY_REALM;KrbHostFQDN=" + IMPALAD_HOST + ";KrbServiceName=impala;SSL=1";
Properties props = new Properties();
props.put("user", userName);
props.put("password", "");
try {
Connection connection = DriverManager.getConnection(jdbcUrl, props);
return connection;
} catch (SQLException e) {
throw new RuntimeException("Fail to create JDBC connection", e);
}
}
private static void testImpalaConnection() throws IOException, SQLException {
int n = 0;
try (Connection cnx = createImpalaConnection()) {
String sql = "show databases";
try (PreparedStatement ps = cnx.prepareStatement(sql)) {
try (ResultSet rs = ps.executeQuery()) {
while (rs.next()) {
n++;
}
}
}
}
logger.debug("{} Impala databases", n);
}
public static void main(String[] args) {
ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
try {
logger.debug("Test Hadoop connection");
testHadoopConnexion();
logger.debug("Hadoop connection - SUCCESS");
} catch (Exception e) {
logger.error("Hadoop connection - FAIL", e);
}
System.out.println();
try {
logger.debug("Test Impala connection");
testImpalaConnection();
logger.debug("Impala connection - SUCCESS");
} catch (Exception e) {
logger.error("Impala connection - FAIL", e);
}
}
}, 1, 10, TimeUnit.SECONDS);
}
}
If the ticket expired or if I perform a "kdestroy" with the command line, Hadoop HDFS listing still works but Impala JDBC driver is unable to get a connection.
Why Impala cannot authenticate whereas Hadoop API calls works ? It the Hadoop API works, it means that the programmatic relogin worked and that a "fresh" ticket is there. Am i right ? Why Impala failed on its side ?
Exception stacktrace
Caused by: java.sql.SQLException: [Simba][ImpalaJDBCDriver](500168) Error creating login context using ticket cache: Unable to obtain Principal Name for authentication .
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_92]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_92]
...
Caused by: com.cloudera.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500168) Error creating login context using ticket cache: Unable to obtain Principal Name for authentication .
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_92]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_92]
...
Caused by: javax.security.auth.login.LoginException: Unable to obtain Principal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:841) ~[?:1.8.0_92]
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704) ~[?:1.8.0_92]
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[?:1.8.0_92]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_92]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_92]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[?:1.8.0_92]
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[?:1.8.0_92]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[?:1.8.0_92]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[?:1.8.0_92]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_92]
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[?:1.8.0_92]
at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[?:1.8.0_92]
at com.cloudera.jdbc.kerberos.Kerberos.getSubjectViaTicketCache(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source) ~[ImpalaJDBC41.jar:ImpalaJDBC_2.5.41.1061]
at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_92]
at java.sql.DriverManager.getConnection(DriverManager.java:208) ~[?:1.8.0_92]
...

Issue when writing to elasticsearch using es-hadoop

Am getting this exception when I'm trying to write to Elasticsearch using mapreduce program with es-hadoop. Am trying to write to index=employee and type=basic which already exists in my Elasticsearch cluster.
My stack trace :-
Exception in thread "main"
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: No resource
['es.resource'] (index/query/location) specified at
org.elasticsearch.hadoop.util.Assert.hasText(Assert.java:30) at
org.elasticsearch.hadoop.mr.EsOutputFormat.init(EsOutputFormat.java:257)
at
org.elasticsearch.hadoop.mr.EsOutputFormat.checkOutputSpecs(EsOutputFormat.java:233)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at
com.mstack.mapreduce.DIGDriver.main(DIGDriver.java:22) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Driver class :-
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "es-hadoop");
job.setJarByClass(DIGDriver.class);
conf.set("es.nodes", "localhost:9200");
conf.set("es.port", "9200");
conf.set("es.resource", "employee/basic");
job.setNumReduceTasks(0);
job.setOutputFormatClass(EsOutputFormat.class);
job.setMapperClass(DIGMapper.class);
job.setMapOutputValueClass(MapWritable.class);
conf.setBoolean("mapreduce.map.speculative", false);
conf.setBoolean("mapreduce.reduce.speculative", false);
boolean status = job.waitForCompletion(true);
if (status) {
System.exit(0);
} else {
System.out.println("Job Failed : Some error!");
System.exit(1);
}
Resolved myself by changing the configs :-
conf.set("es.nodes", "localhost");
conf.set("es.port", "9200");

Hadoop:Cascading FlowException

I Installed hadoop 1.0.4 and hive 0.12.When i run the Cascading Pattern on this it Give Cascading flow exception. when i run with following hadoop command
hadoop jar bulid/libs/pattern-example*.jar
i am getting above mention exception,for reference i include Cascading Code.
Tap inputTap = new Hfs(new TextDelimited(true, "\t"),
"hdfs://hdmaster:54310/user/hive/warehouse/temp/Dataformated/finalformated");
String classifyPath=Output Path;
hdfsPath = classifyPath/pmml File Name;
Tap classifyTap = new Hfs(new TextDelimited(true, "\t"),
classifyPath/pmml File Name));
String formatLocalHdfsData = classifyPath/PMML FILE NAME);
FlowDef flowDef = FlowDef.flowDef().setName("classify")
.addSource("input", inputTap)// input is LFs or HFS
.addSink("classify", classifyTap);
flowDef.addAssemblyPlanner(pmmlPlanner);
Flow classifyFlow = flowConnector.connect(flowDef);
classifyFlow.writeDOT("dot/classify.dot");
classifyFlow.complete();
Cascading Flow Exception
Exception in thread "main" cascading.flow.FlowException: step failed: (1/1) ...eg_Nocoerce20150513093050, with job id: job_201505130921_0003, please see cluster logs for failure messages
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:221)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:149)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Log File Exceprtion
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: cascading.flow.FlowException: internal error during mapper configuration
at cascading.flow.hadoop.FlowMapper.configure(FlowMapper.java:99)
... 14 more
Caused by: java.io.InvalidClassException: cascading.tap.hadoop.Hfs; local class incompatible: stream classdesc serialVersionUID = -2723557385578774808, local class serialVersionUID = -4246440312226820384
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:560)
Please Help Me to solve this Issue.
I resolve the issue. In log file I was getting serialVersionID compatibility issue. Generate the new SerialVersionID and it worked.

Resources