I am developing a java web project which is based on spring. And I want to use Spring JdbcTemplate to connect to hive. But when I tested my service, it came out this error message
"org.springframework.jdbc.CannotGetJdbcConnectionException: Could not
get JDBC Connection; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver
class 'org.apache.hadoop.hive.jdbc.HiveDrive'".
The project is created by idea maven, but the hive jdbc driver is a local jar(it is located at WEB-INF/lib). So I am not sure whether the error is caused by the problem that my project still can't recognize the local jdbc driver jar or just because JdbcTemplate does not support hive connection. Can someone help me figure it out? thank you in advance.
Here are my code:
JdbcTemplate definition:
<bean id="dataSourceTDW" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="${driver}" />
<property name="url" value="${url}" />
<property name="username" value="${username}" />
<property name="password" value="${password}" />
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSourceTDW"/>
</bean>
DAO class:
#Repository(value = "tdwQueryImp")
public class QueryDAOImp implements QueryDAO {
#Autowired
JdbcTemplate jdbcTemplate;
public List<Map<String,Object>> execute(String sql) {
return jdbcTemplate.queryForList(sql);
}
}
Kerberos Example:
#Component
public class HiveDataSource extends SimpleDriverDataSource {
private static final Logger logger = LoggerFactory.getLogger(HiveDataSource.class);
private final Subject subject;
#Autowired
HiveDataSource(Subject subject, Driver hiveDriver, String jdbcUrl) {
this.subject = subject;
setUrl(jdbcUrl);
setDriver(hiveDriver);
}
#Override
protected Connection getConnectionFromDriver(final Properties props) throws SQLException {
try {
return Subject.doAs(subject, (PrivilegedExceptionAction<Connection>)() -> getDriver().connect(getUrl(), props));
} catch (Exception e) {
e.printStackTrace();
logger.error("Failed to get Hive JDBC connection.", e);
}
return null;
}
}
Here HiveDriver bean is defined as :
#Bean
HiveDriver hiveDriver() {
HiveDriver impl = new HiveDriver() {
#Override
public Connection connect(String url, Properties info) throws SQLException {
return acceptsURL(url) ? new HiveConnection(url, info) {
#Override
public void setAutoCommit(boolean autoCommit) throws SQLException {
/* do nothing */
};
} : null;
}
};
return impl;
}
I solved my problem by changing the class of datasource from org.apache.commons.dbcp.BasicDataSource to org.springframework.jdbc.datasource.SimpleDriverDataSource.
Here are the beans configuration:
<bean id="hiveDriver" class="org.apache.hadoop.hive.jdbc.HiveDriver"/>
<bean id="dataSourceTDW" class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
<constructor-arg name="driver" ref="hiveDriver"/>
<constructor-arg name="url" value="${url}"/>
<constructor-arg name="username" value="${username}" />
<constructor-arg name="password" value="${password}" />
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSourceTDW"/>
</bean>
you can refer bellow link :
http://saurzcode.in/2015/01/connect-hiveserver2-service-jdbc-client/
and more
http://hadooptutorial.info/hive-jdbc-client-example/
https://community.hortonworks.com/articles/53629/writing-a-spring-boot-microservices-to-access-hive.html
Related
I am a doing a mid size project with spring jdbc and MsSQL server , project is almost 50% done , now when every request doing lots of inserts and updates specially with those tables which contains lots of columns and large datasets is performing very slow , and sometimes showing connection closed.
Now i am thinking to integrate C3p0 or similar connection pooling but i cant change any DAO code which i already done ..
I implemented a DAOHelper class with JDBCTemplate variable and injecting the JDBCTemplate dependency in applicationContext.xml with autowiring of DAOClass in controller class , and i extended this DAOHelper to all DAO classes and using this jdbcTemplate to do JDBC operations.
<bean id="ds" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
<property name="url" value="jdbc:sqlserver://192.168.1.101:1433;databaseName=OrderManager"/>
<property name="username" value="sa"/>
<property name="password" value="520759"/>
</bean>
<bean id="JdbcDataSource" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="ds"/>
</bean>
<bean id="OrderDAO" class="com.ordermanager.order.dao.OrderDAO" >
<property name="jdbcTemplate" ref="JdbcDataSource"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
#Controller
public class OrderController {
#Autowired
OrderDAO orderDAO;
#RequestMapping(value = "/addNewItem", method = RequestMethod.GET)
public ModelAndView addItem(#RequestParam("ParamData") JSONObject paramJson) {
ApplicationContext ctx = new FileSystemXmlApplicationContext(ConstantContainer.Application_Context_File_Path);
OrderDAO orderDAO = (OrderDAO) ctx.getBean("OrderDAO");
return new ModelAndView("MakeResponse", "responseValue", orderDAO.addItem(paramJson));
}
public class DAOHelper {
private JdbcTemplate jdbcTemplate;
private PlatformTransactionManager transactionManager;
public PlatformTransactionManager getTransactionManager() {
return transactionManager;
}
public void setTransactionManager(PlatformTransactionManager txManager) {
this.transactionManager = txManager;
}
public JdbcTemplate getJdbcTemplate() /*I am using this Method for all JDBC Task*/ {
return jdbcTemplate;
}
public void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
Now with minimal code changes how can i integrate C3p0 or any good connection pooling library with my already written code.
Just change the ds bean in your config xml with following and consider adding other c3p0 properties according to your own. make sure to have c3p0 jar in your classpath.
<bean id="ds" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
<property name="jdbcUrl" value="jdbc:sqlserver://192.168.1.101:1433;databaseName=OrderManager" />
<property name="user" value="sa" />
<property name="password" value="520789" />
</bean>
Log4j2 JDBC appender can be setup using a pooled connection factory that is defined using calls and method (see log4j2 Appenders):
<ConnectionFactory class="net.example.db.ConnectionFactory" method="getDatabaseConnection" />
Using Spring I have already a defined datasource that is providing a pooled connection :
<bean id="myDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${db_driver_class}" />
<property name="url" value="${db_jdbc_url}" />
<property name="username" value="${db_username}" />
<property name="password" value="${db_password}" />
<property name="initialSize" value="10" />
<property name="maxActive" value="100" />
<property name="maxIdle" value="50" />
<property name="minIdle" value="10" />
<property name="validationQuery" value="select 1" />
<property name="testOnBorrow" value="true" />
I would like to use the the Spring connection pool for the JDBC appender. Any idea how can this be done ?
thanks
Raz
I mange to create a 3-steps solution :
Define a bean in spring context that provide access to a data
source
Build an implementation of the bean that provide the
desired connection.
Build a static wrapper that can be accessed by the log4j JDBC appender.
1st step - bean declaration :
<bean id="springConnection" class="com.dal.entities.SpringConnection" scope="singleton">
<property name="dataSource" ref="myDataSource" />
the 2nd step - bean implementation - is also simple :
class SpringConnection {
private DataSource dataSource;
public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
}
Connection getConnection() throws Exception {
return dataSource.getConnection();
}
}
the 3rd part - wrapper with static method - is a bit more complex:
public class SpringAccessFactory {
private final SpringConnection springCon;
private static ApplicationContext context;
private interface Singleton {
final SpringAccessFactory INSTANCE = new SpringAccessFactory();
}
private SpringAccessFactory() {
this.springCon = context.getBean(SpringConnection.class);
}
public static Connection getConnection() throws Exception {
return Singleton.INSTANCE.springCon.getConnection();
}
public static void setContext( ApplicationContext context) {
SpringAccessFactory.context = context;
}
}
There are - however - 2 issues I found so far:
You need to initialize the spring context and send it into the wrapper (SpringAccessFactory.setConetxt) before you start using the logger
initializing spring context early in program may trigger #PostConstruct methods (if any exists), before you plan to do so.....
Our current development based on Discriminator model in Multi-tenancy. Following is the technical stack we are currently engage with,
Spring 3.1.1.RELEASE
Hibernate 4.1.6.Final
We are maintain tenant id by keep one column separately in each table. Tenant id filter when session is created.
Example model class.
#Entity
#FilterDef(name = "tenantFilter", parameters = #ParamDef(name = "tenantIdParam", type = "string"))
#Filters(#Filter(name = "tenantFilter", condition = "tenant_id = :tenantIdParam"))
#Table(name = "assessment")
public class Assessment implements java.io.Serializable, Comparable<Assessment> {
private static final long serialVersionUID = -2231966582754667029L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "id", unique = true, nullable = false)
private Long id;
#Column(name = "tenant_id", nullable = false, updatable = false)
private String tenantId;
// rest of the code...
}
This is the configuration of session factory
<!-- create database connection pool -->
<bean id="dataSource" class="com.jolbox.bonecp.BoneCPDataSource" destroy-method="close">
<property name="driverClass" value="${jdbc.driverClassName}" />
<property name="jdbcUrl" value="jdbc:mysql://${jdbc.host}:3306/${jdbc.database}?createDatabaseIfNotExist=true&autoReconnect=true&useUnicode=true&connectionCollation=utf8_general_ci&characterEncoding=UTF-8" />
<property name="username" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="idleConnectionTestPeriodInMinutes" value="60"/>
<property name="idleMaxAgeInMinutes" value="240"/>
<property name="maxConnectionsPerPartition" value="30"/>
<property name="minConnectionsPerPartition" value="10"/>
<property name="partitionCount" value="3"/>
<property name="acquireIncrement" value="5"/>
<property name="statementsCacheSize" value="100"/>
<property name="releaseHelperThreads" value="3"/>
</bean>
<!-- Hibernate SessionFactory -->
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="packagesToScan" value="lk.gov.elg.orm.model"/>
<property name="hibernateProperties">
<value>
hibernate.dialect=${hibernate.dialect}
hibernate.hbm2ddl.auto=update
</value>
</property>
</bean>
<bean id="tenantBasedSessionFactory" class="lk.gov.elg.orm.dao.impl.TenantBasedSessionFactoryImpl">
<property name="sessionFactory" ref="sessionFactory"/>
</bean>
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory"/>
</bean>
Tenant base session factory
public class TenantBasedSessionFactoryImpl implements TenantBasedSessionFactory {
private SessionFactory sessionFactory;
#Override
public Session getTenantBasedSession(Object tenantId) {
Session session = sessionFactory.openSession();
session.enableFilter("tenantFilter").setParameter("tenantIdParam", tenantId);
return session;
}
public void setSessionFactory(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
public Session getAllTenantBasedSession() {
Session session = sessionFactory.openSession();
return session;
}
}
sample Service class
#Service("assessmentService")
public class AssessmentServiceImpl implements AssessmentService {
#Autowired
private AssessmentDao assessmentDao;
public Long saveAssessment(Assessment assessment, Object tenantId) {
return assessmentDao.saveAssessment(assessment, tenantId);
}
}
Sample DAO class
#Repository("assessmentDao")
public class AssessmentDaoImpl implements AssessmentDao {
#Autowired
private TenantBasedSessionFactory tenantBasedSessionFactory;
public Long saveAssessment(Assessment assessment, Object tenantId) {
Session session = tenantBasedSessionFactory.getTenantBasedSession(tenantId);
try {
session.beginTransaction();
session.save(assessment);
session.getTransaction().commit();
return assessment.getId();
} catch (HibernateException e) {
logger.error("Error in persist assessment:", e);
session.getTransaction().rollback();
return null;
} finally {
session.close();
}
}
}
I would like to know is there a way to get the spring transaction support with this Discriminator model for database transactions ?
And the other thing is I would like to know is there any advantage of give transaction handling to spring rather than handling it by our side?
Thanks in advance.
i had similar problem and i have resolved it using aspect instead of customizing sessionfactory, so i can leverage annotation driven transaction support
Below code is for aspect which works on annotation #Tennant
#Aspect
public class TennantAspect {
#Autowired
private SessionFactory sessionFactory;
#Around("#annotation(Tennant)")
public Object enableClientFilter(ProceedingJoinPoint pjp) throws Throwable {
Object obj;
boolean isDAO=(pjp.getTarget() instanceof BaseDAO<?,?>);
try {
if(isDAO){
Authentication auth=SecurityContextHolder.getContext().getAuthentication();
if(auth!=null){
User user=(User) auth.getPrincipal();
this.sessionFactory.getCurrentSession().enableFilter("clientFilter").setParameter("clientId", user.getClientId());
}
}
obj=pjp.proceed();
}finally {
if(isDAO)
this.sessionFactory.getCurrentSession().disableFilter("clientFilter");
}
return obj;
}
}
Hope this solves your problem.
Alternatively you can also look at tenancy support by hiberante & spring
http://docs.jboss.org/hibernate/orm/4.1/devguide/en-US/html/ch16.html
https://github.com/mariofts/spring-multitenancy
I am trying to use mybatis spring transaction management
My problem is that the transactions are getting committed even if an exception is thrown.
Relatively new to this, anykind of help is much appreciated.
Following are the code snippets
spring xml configuration
<bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location">
<value>classpath:Config.properties</value>
</property>
</bean>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${db.driver}"/>
<property name="url" value="${db.url}"/>
<property name="username" value="${db.user}"/>
<property name="password" value="${db.pass}"/>
<property name="defaultAutoCommit" value="false" />
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="configLocation" value="classpath:Configuration.xml" />
<property name="dataSource" ref="dataSource" />
</bean>
<bean class="org.mybatis.spring.SqlSessionTemplate" id="sqlSessionTemplate">
<constructor-arg ref="sqlSessionFactory"/>
</bean>
Service class
#Transactional(rollbackFor=Exception.class, propagation=Propagation.REQUIRED)
public void insertNotes(String noteTypeId,String confidentialValue,String summaryValue,String notes ,String notesId,String noteTypeValue,
String claimNumber,String notepadId,String mode)
{
NotepadExample notepadExample= new NotepadExample();
//to be moved into dao class marked with transaction boundaries
Notepad notepad = new Notepad();
notepad.setAddDate(new Date());
notepad.setAddUser("DummyUser");
if("true".equalsIgnoreCase(confidentialValue))
confidentialValue="Y";
else
confidentialValue="N";
notepad.setConfidentiality(confidentialValue);
Long coverageId=getCoverageId(claimNumber);
notepad.setCoverageId(coverageId);
notepad.setDescription(summaryValue);
notepad.setEditUser("DmyEditUsr");
//notepad.setNotepadId(new Long(4)); //auto sequencing
System.out.println(notes);
notepad.setNotes(notes);
notepad.setNoteType(noteTypeValue); //Do we really need this?
notepad.setNoteTypeId(Long.parseLong(notesId));
if("update".equalsIgnoreCase(mode))
{
notepad.setNotepadId(new Long(notepadId));
notepad.setEditDate(new Date());
notepadMapper.updateByPrimaryKeyWithBLOBs(notepad);
}
else
notepadMapper.insertSelective(notepad);
throw new java.lang.UnsupportedOperationException();
}
Not sure where I am going wrong...
The current call is from the controller as given below
#Controller
public class NotesController {
private static final Logger logger = LoggerFactory
.getLogger(NotesController.class);
#Autowired
private Utils utility;
#Autowired
NotepadService notepadService;
public #ResponseBody List<? extends Object> insertNotes(HttpServletRequest request,
HttpServletResponse response,#RequestParam("noteTypeValue") String noteTypeId,
#RequestParam("confidentialValue")String confidentialValue,
#RequestParam("summaryValue")String summaryValue,
#RequestParam("notes")String notes ,
#RequestParam("notesId")String notesId,
#RequestParam("noteTypeValue")String noteTypeValue,
#RequestParam("claimNumber")String claimNumber,
#RequestParam("notepadId")String notepadId,
#RequestParam("mode")String mode) {
try {
notepadService.insertNotes(noteTypeId, confidentialValue, summaryValue, notes, notesId, noteTypeValue, claimNumber, notepadId, mode);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
}
I had the same issue. I am also relatively new to spring. But according to me it depends on how you are calling your insertNotes() method. If you are calling it from another local method then it will not work, because spring has no way of know that it is called and to start the transaction.
If you are calling it from a method of another class by using autowired object of the class which contains insertNotes() method, then it should work.
For example
class ABC
{
#Autowired
NotesClass notes;
public void testMethod() {
notes.insertNotes();
}
}
class NotesClass
{
#Transactional(rollbackFor=Exception.class, propagation=Propagation.REQUIRED)
public void insertNotes(String noteTypeId,
String confidentialValue,
String summaryValue,String notes ,
String notesId,String noteTypeValue,
String claimNumber,
String notepadId,
String mode) {
//Your code
}
}
You can try using transaction template. Remove #Tranasactional annotation from method and following code to xml file.
<bean id="trTemplate" class="org.springframework.transaction.support.TransactionTemplate">
<property name="timeout" value="30"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
Create object of Trasactiontemplate and call insertNotes from controller like this
#Autowired
private TransactionTemplate transactionTemplate;
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
#Override
protected void doInTransactionWithoutResult(TransactionStatus transactionStatus) {
try {
insertNotes();
} catch (Exception e) {
transactionStatus.setRollbackOnly();
logger.error("Exception ocurred when calling insertNotes", e);
throw new RuntimeException(e);
}
}
});
Note : You have to make all parameters final before calling insertNotes method
We have an application where we have used spring for IOC. We have the dataSource bean configured in applicationContext.xml and that is referenced in other bean definations.
The dataSource bean defination looks like:
<bean id="dbDataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="url"
value="jdbc:oracle:oci:#TESTDB" />
<property name="username" value="TESTUSER" />
<property name="password" value="TESTPWD" />
<property name="initialSize" value="50" />
<property name="maxActive" value="40" />
<property name="maxIdle" value="10" />
<property name="minIdle" value="10" />
<property name="maxWait" value="-1" />
</bean>
<bean id="serviceDAO" class="com.test.impl.ServiceDAOImpl">
<property name="dataSource" ref="dbDataSource" />
</bean>
ServiceDAOImpl looks as follows:
public class ServiceDAOImpl implements ServiceDAO {
private JdbcTemplate jdbcTemplate;
public void setDataSource(DataSource dataSource) {
this.jdbcTemplate = new JdbcTemplate(dataSource);
}
#SuppressWarnings({ "rawtypes", "unchecked" })
public ValueObj readValue(String key) {
String query = "SELECT * FROM SERVICE_LOOKUP WHERE KEY=?";
/**
* Implement the RowMapper callback interface
*/
return (ValueObj) jdbcTemplate.queryForObject(query,
new Object[] { key }, new RowMapper() {
public Object mapRow(ResultSet resultSet, int rowNum)
throws SQLException {
return new ValueObj(resultSet.getString("KEY"),
resultSet.getString("VALUE"));
}
});
}
public ServiceDAOImpl() {
}
}
Now, at the server start up injection is happening fine and when we use the dataSource in serviceDAOImpl the connection is happening fine. But the very first time the database call is made it takes around 3 mins to get the response back. I think this is because the pool creation is done during the first call and we have set the parameter "initialSize" = 50 in applicationConext.xml.
So, to avoid this we need a way in which the pool can be created during the application startup itself and can be used directly.
Please suggest. Let me know if any clarification required.
Regards
Saroj
There's a work-around for this .You could force jdbcTemplate to use the
DB connection at startup. See the link here for detailed explanation .
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<constructor-arg index="0" ref="dataSource"/>
<constructor-arg index="1" value="false"/>
</bean>
The second constructor-arg is the lazy Init flag.
Aravind A's solution is the preffered one, but just in case you don't want to define an extra bean you can point spring to your DAO's init method:
<bean id="serviceDAO" class="com.test.impl.ServiceDAOImpl" init-method="init">
<property name="dataSource" ref="dbDataSource" />
</bean>
and then define ServiceDAOImpl.init() which calls some sql like SELECT 1 FROM SERVICE_LOOKUP LIMIT 1 or even better some noop like SELECT 1:
public class ServiceDAOImpl implements ServiceDAO {
public void init() {
String query = "SELECT 1 FROM SERVICE_LOOKUP LIMIT 1";
int i = jdbcTemplate.queryForInt(query);
}
}