log4j2 JDBC appender with Spring - spring

Log4j2 JDBC appender can be setup using a pooled connection factory that is defined using calls and method (see log4j2 Appenders):
<ConnectionFactory class="net.example.db.ConnectionFactory" method="getDatabaseConnection" />
Using Spring I have already a defined datasource that is providing a pooled connection :
<bean id="myDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${db_driver_class}" />
<property name="url" value="${db_jdbc_url}" />
<property name="username" value="${db_username}" />
<property name="password" value="${db_password}" />
<property name="initialSize" value="10" />
<property name="maxActive" value="100" />
<property name="maxIdle" value="50" />
<property name="minIdle" value="10" />
<property name="validationQuery" value="select 1" />
<property name="testOnBorrow" value="true" />
I would like to use the the Spring connection pool for the JDBC appender. Any idea how can this be done ?
thanks
Raz

I mange to create a 3-steps solution :
Define a bean in spring context that provide access to a data
source
Build an implementation of the bean that provide the
desired connection.
Build a static wrapper that can be accessed by the log4j JDBC appender.
1st step - bean declaration :
<bean id="springConnection" class="com.dal.entities.SpringConnection" scope="singleton">
<property name="dataSource" ref="myDataSource" />
the 2nd step - bean implementation - is also simple :
class SpringConnection {
private DataSource dataSource;
public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
}
Connection getConnection() throws Exception {
return dataSource.getConnection();
}
}
the 3rd part - wrapper with static method - is a bit more complex:
public class SpringAccessFactory {
private final SpringConnection springCon;
private static ApplicationContext context;
private interface Singleton {
final SpringAccessFactory INSTANCE = new SpringAccessFactory();
}
private SpringAccessFactory() {
this.springCon = context.getBean(SpringConnection.class);
}
public static Connection getConnection() throws Exception {
return Singleton.INSTANCE.springCon.getConnection();
}
public static void setContext( ApplicationContext context) {
SpringAccessFactory.context = context;
}
}
There are - however - 2 issues I found so far:
You need to initialize the spring context and send it into the wrapper (SpringAccessFactory.setConetxt) before you start using the logger
initializing spring context early in program may trigger #PostConstruct methods (if any exists), before you plan to do so.....

Related

Can Spring JdbcTemplate connect to hive?

I am developing a java web project which is based on spring. And I want to use Spring JdbcTemplate to connect to hive. But when I tested my service, it came out this error message
"org.springframework.jdbc.CannotGetJdbcConnectionException: Could not
get JDBC Connection; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver
class 'org.apache.hadoop.hive.jdbc.HiveDrive'".
The project is created by idea maven, but the hive jdbc driver is a local jar(it is located at WEB-INF/lib). So I am not sure whether the error is caused by the problem that my project still can't recognize the local jdbc driver jar or just because JdbcTemplate does not support hive connection. Can someone help me figure it out? thank you in advance.
Here are my code:
JdbcTemplate definition:
<bean id="dataSourceTDW" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="${driver}" />
<property name="url" value="${url}" />
<property name="username" value="${username}" />
<property name="password" value="${password}" />
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSourceTDW"/>
</bean>
DAO class:
#Repository(value = "tdwQueryImp")
public class QueryDAOImp implements QueryDAO {
#Autowired
JdbcTemplate jdbcTemplate;
public List<Map<String,Object>> execute(String sql) {
return jdbcTemplate.queryForList(sql);
}
}
Kerberos Example:
#Component
public class HiveDataSource extends SimpleDriverDataSource {
private static final Logger logger = LoggerFactory.getLogger(HiveDataSource.class);
private final Subject subject;
#Autowired
HiveDataSource(Subject subject, Driver hiveDriver, String jdbcUrl) {
this.subject = subject;
setUrl(jdbcUrl);
setDriver(hiveDriver);
}
#Override
protected Connection getConnectionFromDriver(final Properties props) throws SQLException {
try {
return Subject.doAs(subject, (PrivilegedExceptionAction<Connection>)() -> getDriver().connect(getUrl(), props));
} catch (Exception e) {
e.printStackTrace();
logger.error("Failed to get Hive JDBC connection.", e);
}
return null;
}
}
Here HiveDriver bean is defined as :
#Bean
HiveDriver hiveDriver() {
HiveDriver impl = new HiveDriver() {
#Override
public Connection connect(String url, Properties info) throws SQLException {
return acceptsURL(url) ? new HiveConnection(url, info) {
#Override
public void setAutoCommit(boolean autoCommit) throws SQLException {
/* do nothing */
};
} : null;
}
};
return impl;
}
I solved my problem by changing the class of datasource from org.apache.commons.dbcp.BasicDataSource to org.springframework.jdbc.datasource.SimpleDriverDataSource.
Here are the beans configuration:
<bean id="hiveDriver" class="org.apache.hadoop.hive.jdbc.HiveDriver"/>
<bean id="dataSourceTDW" class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
<constructor-arg name="driver" ref="hiveDriver"/>
<constructor-arg name="url" value="${url}"/>
<constructor-arg name="username" value="${username}" />
<constructor-arg name="password" value="${password}" />
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSourceTDW"/>
</bean>
you can refer bellow link :
http://saurzcode.in/2015/01/connect-hiveserver2-service-jdbc-client/
and more
http://hadooptutorial.info/hive-jdbc-client-example/
https://community.hortonworks.com/articles/53629/writing-a-spring-boot-microservices-to-access-hive.html

Connection pool replacement for already implemented Spring Jdbctemplate project

I am a doing a mid size project with spring jdbc and MsSQL server , project is almost 50% done , now when every request doing lots of inserts and updates specially with those tables which contains lots of columns and large datasets is performing very slow , and sometimes showing connection closed.
Now i am thinking to integrate C3p0 or similar connection pooling but i cant change any DAO code which i already done ..
I implemented a DAOHelper class with JDBCTemplate variable and injecting the JDBCTemplate dependency in applicationContext.xml with autowiring of DAOClass in controller class , and i extended this DAOHelper to all DAO classes and using this jdbcTemplate to do JDBC operations.
<bean id="ds" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
<property name="url" value="jdbc:sqlserver://192.168.1.101:1433;databaseName=OrderManager"/>
<property name="username" value="sa"/>
<property name="password" value="520759"/>
</bean>
<bean id="JdbcDataSource" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="ds"/>
</bean>
<bean id="OrderDAO" class="com.ordermanager.order.dao.OrderDAO" >
<property name="jdbcTemplate" ref="JdbcDataSource"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
#Controller
public class OrderController {
#Autowired
OrderDAO orderDAO;
#RequestMapping(value = "/addNewItem", method = RequestMethod.GET)
public ModelAndView addItem(#RequestParam("ParamData") JSONObject paramJson) {
ApplicationContext ctx = new FileSystemXmlApplicationContext(ConstantContainer.Application_Context_File_Path);
OrderDAO orderDAO = (OrderDAO) ctx.getBean("OrderDAO");
return new ModelAndView("MakeResponse", "responseValue", orderDAO.addItem(paramJson));
}
public class DAOHelper {
private JdbcTemplate jdbcTemplate;
private PlatformTransactionManager transactionManager;
public PlatformTransactionManager getTransactionManager() {
return transactionManager;
}
public void setTransactionManager(PlatformTransactionManager txManager) {
this.transactionManager = txManager;
}
public JdbcTemplate getJdbcTemplate() /*I am using this Method for all JDBC Task*/ {
return jdbcTemplate;
}
public void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
Now with minimal code changes how can i integrate C3p0 or any good connection pooling library with my already written code.
Just change the ds bean in your config xml with following and consider adding other c3p0 properties according to your own. make sure to have c3p0 jar in your classpath.
<bean id="ds" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
<property name="jdbcUrl" value="jdbc:sqlserver://192.168.1.101:1433;databaseName=OrderManager" />
<property name="user" value="sa" />
<property name="password" value="520789" />
</bean>

How to inject a dependency bean to GridCacheStore implementation?

My config:
<bean parent="cache-template">
<property name="name" value="yagoLabel" />
<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="distributionMode" value="PARTITIONED_ONLY" />
<property name="backups" value="1" />
<property name="store">
<bean class="id.ac.itb.ee.lskk.lumen.yago.YagoLabelCacheStore" autowire="byType" init-method="init" />
</property>
<property name="writeBehindEnabled" value="true" />
<property name="writeBehindFlushSize" value="102380" />
<property name="writeBehindFlushFrequency" value="30000" />
<property name="writeBehindBatchSize" value="10240" />
<property name="swapEnabled" value="false" />
<property name="evictionPolicy">
<bean class="org.gridgain.grid.cache.eviction.lru.GridCacheLruEvictionPolicy">
<property name="maxSize" value="102400" />
</bean>
</property>
</bean>
And I start GridGain as follows:
My GridCacheStore implementation:
public class YagoLabelCacheStore extends GridCacheStoreAdapter<String, YagoLabel> {
private static final Logger log = LoggerFactory
.getLogger(YagoLabelCacheStore.class);
private DBCollection labelColl;
#GridSpringResource(resourceName="mongoDb")
private DB db;
#Inject
private GridGainSpring grid;
#PostConstruct
public void init() {
log.info("Grid is {}", grid);
labelColl = db.getCollection("label");
}
I start GridGain as follows:
String entityId = "Muhammad";
try (AnnotationConfigApplicationContext appCtx
= new AnnotationConfigApplicationContext(LumenConfig.class)) {
Grid grid = appCtx.getBean(Grid.class);
GridCache<String, YagoLabel> labelCache = YagoLabel.cache(grid);
log.info("Label for {}: {}", entityId, labelCache.get(entityId));
}
LumenConfig Spring configuration contains a DB bean named mongoDb.
However this throws NullPointerException because db is not injected properly. I tried #Inject GridGainSpring just for testing, and even GridGainSpring itself is not injected.
I also tried setting <property name="db" ref="mongoDb"/> in the GridGain Config XML but Spring complains cannot find the bean.
My workaround is to put it inside a public static field but that's soo hacky: https://github.com/ceefour/lumen-kb/blob/b8445fbebd227fb7ac337c758a60badb7ecd3095/cli/src/main/java/id/ac/itb/ee/lskk/lumen/yago/YagoLabelCacheStore.java
The way is to load the GridConfiguration using Spring, then pass it to GridGainSpring.start() :
// "classpath:" is required, otherwise it won't be found in a WAR
#ImportResource("classpath:id/ac/itb/ee/lskk/lumen/core/lumen.gridgain.xml")
#Configuration
public static class GridGainConfig {
#Inject
private ApplicationContext appCtx;
#Inject
private GridConfiguration gridCfg;
#Bean(destroyMethod="close")
public Grid grid() throws GridException {
return GridGainSpring.start(gridCfg, appCtx);
}
}
:-)

DbUnit H2 in memory db with Spring and Hibernate

Hi I'm trying a little POC with JPA and unit test to verify that the DB schema is created. I'm working with H2 DB and I set to Hibernate create the schema from the entities, but when DbUnit tries to initialize the DB from a dataset I always get a Table ... not found in tableMap. I read that I have to add the property DB_CLOSE_DELAY=-1 to DB URL but is like after Hibernate creates the schema the DB is losted when DbUnit tries to initialize.
Any ideas? Any help is highly appreciated.
This is my config:
application-context.xml
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSourceH2" />
<property name="packagesToScan" value="com.xxx.model" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="generateDdl" value="true" />
<property name="showSql" value="true" />
<!-- property name="databasePlatform" value="org.hibernate.dialect.MySQLInnoDBDialect" /-->
<property name="databasePlatform" value="org.hibernate.dialect.H2Dialect" />
<!-- property name="database" value="MYSQL" /-->
</bean>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.hbm2ddl.auto">create</prop>
<prop key="javax.persistence.validation.mode">CALLBACK</prop>
</props>
</property>
</bean>
<bean id="dataSourceH2"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.h2.Driver" />
<property name="url" value="jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1" />
<property name="username" value="sa" />
<property name="password" value="" />
</bean>
RepositoryTest.java
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = { "/application-context-test.xml" })
#Transactional
public class SystemEntityRepositoryH2Test {
#Inject
private SystemEntityRepository repository;
#Inject
private DataSource dataSourceH2;
private IDatabaseConnection connection;
#Before
public void setUp() throws Exception {
IDatabaseConnection dbUnitCon = null;
dbUnitCon = new DatabaseDataSourceConnection(dataSourceH2, "testdb");
dbUnitCon.getConfig().setProperty(DatabaseConfig.FEATURE_QUALIFIED_TABLE_NAMES, true);
IDataSet dataSet = this.getDataSet("dataset-systementity.xml");
DatabaseOperation.INSERT.execute(dbUnitCon, dataSet);
}
#After
public void tearDown() throws Exception {
//DatabaseOperation.DELETE_ALL.execute(this.getConnection(), this.getDataSet(dataSetFile));
}
#Test
public void test() throws Exception {
}
protected IDataSet getDataSet(String dataSetFile) throws Exception {
ResourceLoader resourceLoader = new ClassRelativeResourceLoader(this.getClass());
Resource resource = resourceLoader.getResource(dataSetFile);
if (resource.exists()) {
return new FlatXmlDataSetBuilder().build(resource.getInputStream());
}
return null;
}
}
dataset-systementity.xml
<?xml version="1.0" encoding="UTF-8"?>
<dataset>
<System_Entities id="2" name="NAME" phone01="+52-55-55555555" email="a#a.com"
address01="Street" address02="123" address03="1" address04="Address04"
address05="Address05" city="City" state="State" country="MX"
zipcode="12345" />
</dataset>
Error
ERROR DatabaseDataSet:286 - Table 'System_Entities' not found in tableMap=org.dbunit.dataset.OrderedTableNameMap[_tableNames=[], _tableMap={}, _caseSensitiveTableNames=false]
I can see that the tables are created by hibernate because the log shows all the sql sentences without error.
Thanks.
SOLUTION
Thanks Mark Robinson
I modified the setUp method to:
#Before
public void setUp() throws Exception {
IDatabaseConnection dbUnitCon = null;
EntityManager entityManager = entityManagerFactory.createEntityManager();
Session session = entityManager.unwrap(Session.class);
SessionImplementor si = (SessionImplementor) session;
Connection conn = si.getJdbcConnectionAccess().obtainConnection();
dbUnitCon = new DatabaseConnection(conn);
//dbUnitCon.getConfig().setProperty(DatabaseConfig.FEATURE_QUALIFIED_TABLE_NAMES, true);
IDataSet dataSet = this.getDataSet("dataset-systementity.xml");
DatabaseOperation.INSERT.execute(dbUnitCon, dataSet);
}
It works now, what I don't understand yet is if I use HSQLDB I don't have this problem.
The problem is that DBUnit is loading the table data before Hibernate can initialize.
As part of your #setup, you'll need to get the Hibernate session. This should cause Hibernate to create your table. You could even force it by executing a simple query like select 1

Datasource initialization at server start up

We have an application where we have used spring for IOC. We have the dataSource bean configured in applicationContext.xml and that is referenced in other bean definations.
The dataSource bean defination looks like:
<bean id="dbDataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="url"
value="jdbc:oracle:oci:#TESTDB" />
<property name="username" value="TESTUSER" />
<property name="password" value="TESTPWD" />
<property name="initialSize" value="50" />
<property name="maxActive" value="40" />
<property name="maxIdle" value="10" />
<property name="minIdle" value="10" />
<property name="maxWait" value="-1" />
</bean>
<bean id="serviceDAO" class="com.test.impl.ServiceDAOImpl">
<property name="dataSource" ref="dbDataSource" />
</bean>
ServiceDAOImpl looks as follows:
public class ServiceDAOImpl implements ServiceDAO {
private JdbcTemplate jdbcTemplate;
public void setDataSource(DataSource dataSource) {
this.jdbcTemplate = new JdbcTemplate(dataSource);
}
#SuppressWarnings({ "rawtypes", "unchecked" })
public ValueObj readValue(String key) {
String query = "SELECT * FROM SERVICE_LOOKUP WHERE KEY=?";
/**
* Implement the RowMapper callback interface
*/
return (ValueObj) jdbcTemplate.queryForObject(query,
new Object[] { key }, new RowMapper() {
public Object mapRow(ResultSet resultSet, int rowNum)
throws SQLException {
return new ValueObj(resultSet.getString("KEY"),
resultSet.getString("VALUE"));
}
});
}
public ServiceDAOImpl() {
}
}
Now, at the server start up injection is happening fine and when we use the dataSource in serviceDAOImpl the connection is happening fine. But the very first time the database call is made it takes around 3 mins to get the response back. I think this is because the pool creation is done during the first call and we have set the parameter "initialSize" = 50 in applicationConext.xml.
So, to avoid this we need a way in which the pool can be created during the application startup itself and can be used directly.
Please suggest. Let me know if any clarification required.
Regards
Saroj
There's a work-around for this .You could force jdbcTemplate to use the
DB connection at startup. See the link here for detailed explanation .
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<constructor-arg index="0" ref="dataSource"/>
<constructor-arg index="1" value="false"/>
</bean>
The second constructor-arg is the lazy Init flag.
Aravind A's solution is the preffered one, but just in case you don't want to define an extra bean you can point spring to your DAO's init method:
<bean id="serviceDAO" class="com.test.impl.ServiceDAOImpl" init-method="init">
<property name="dataSource" ref="dbDataSource" />
</bean>
and then define ServiceDAOImpl.init() which calls some sql like SELECT 1 FROM SERVICE_LOOKUP LIMIT 1 or even better some noop like SELECT 1:
public class ServiceDAOImpl implements ServiceDAO {
public void init() {
String query = "SELECT 1 FROM SERVICE_LOOKUP LIMIT 1";
int i = jdbcTemplate.queryForInt(query);
}
}

Resources