Spring Batch- Xml based configuration performance over Java based - spring

I am trying to convert the spring batch configuration from xml based to annotation based.
Below is my xml based configuration.
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- Step will need a transaction manager -->
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="dbMapper" class="org.test.DBValueMapper">
</bean>
<bean id="dbMapperFlatfile" class="org.test.FlatFileRowMapper">
</bean>
<bean id="paramSetter" class="org.test.DBParamSetter">
</bean>
<bean id="dbReader" class="org.test.DBValueReader"
scope="step">
<property name="paramSetter" ref="paramSetter"/>
<property name="verifyCursorPosition" value="false" />
<property name="dataSource" ref="dataSource" />
<property name="sql" value="#{jobParameters['SQL_QUERY']}" />
<property name="rowMapper" ref="dbMapper" />
<property name="fetchSize" value="5000" />
</bean>
<bean id="dbWriterIO" class="org.test.TemplateWritterIO"
scope="step">
<property name="velocityEngine" ref="velocityEngine" />
<!-- <property name="rptConfig" value="#{jobParameters['RPT_CONFIGVAL']}" /> -->
<property name="headerCallback" ref="dbWriterIO" />
<property name="footerCallback" ref="dbWriterIO" />
</bean>
<batch:job id="fileGenJobNio">
<batch:step id="fileGenJobStempNio">
<batch:tasklet>
<batch:chunk reader="dbReader" writer="dbWriterNIO"
commit-interval="5000">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
Below is the equivalent Java based configuration:
#EnableBatchProcessing
#Import({ServiceConfiguration.class})
public class SRBatchGenerator extends DefaultBatchConfigurer{
#Autowired
private JobBuilderFactory jobBuilders;
#Autowired
private StepBuilderFactory stepBuilders;
#Autowired
private VelocityEngine velocityEngine;
#Autowired
private DBValueMapper mapper;
#Autowired
private DbHelper dbhelper;
#Autowired
private DataSource datasource;
#Bean
public Step step(){
return stepBuilders.get("step")
.chunk(5000)
.reader(reader())
//.processor(processor())
.writer(writer())
//.listener(logProcessListener())
.faultTolerant()
//.skipLimit(10)
//.skip(UnknownGenderException.class)
//.listener(logSkipListener())
.build();
}
#Bean
public Job fileGeneratorJob(){
return jobBuilders.get("fileGeneratorJob")
//.listener(protocolListener())
.start(step())
.build();
}
#Bean
public DBValueMapper mapper(){
return new DBValueMapper();
}
#Bean
#StepScope
public DBValueReader3 reader(){
String query="Select Test1,Test2,test3,test4 from RPt_TEST";
DBValueReader3 dbread = new DBValueReader3();
dbread.setSql(query);
dbread.setRowMapper(mapper);
dbread.setDataSource(datasource);
return dbread;
}
#Bean
#StepScope
public TemplateWritterIO writer(){
TemplateWritterIO writer=new TemplateWritterIO();
writer.setVelocityEngine(velocityEngine);
return writer;
}
#Override
protected JobRepository createJobRepository() throws Exception {
MapJobRepositoryFactoryBean factory =
new MapJobRepositoryFactoryBean();
factory.afterPropertiesSet();
return (JobRepository) factory.getObject();
}
}
When I execute my Job using xml based it took 27sec to write 1 Million record into Flat file.
But to write same 1 million record, Java based job took about 2 hours to write.
I am not sure what I am missing here. Can anyone help me or guide me why it is slow in Java based configuration.

Related

Spring configuration split between xml resource and Java configuration

I am trying to mix both xml and Java Configuration.
I have a spring-security.xml resource that I import in my application boot.
Say this was a part of the initial xml:
<bean id="ldapContextSource" class="org.springframework.ldap.core.support.LdapContextSource">
<property name="url" value="${ldap.url}" />
<property name="base" value="${ldap.base}" />
<property name="userDn" value="${ldap.user}" />
<property name="password" value="${ldap.password}" />
</bean>
<bean id="ldapTemplate" class="org.springframework.ldap.core.LdapTemplate">
<constructor-arg ref="ldapContextSource" />
</bean>
Can I move just this part to be a Java config ? Or would the references be an issue.
Thank You
You can move it to Java Config
Declare configuration class
#Configuration
public class AppConfig {
#Bean
public LdapContextSource ldapContextSource(){
LdapContextSource lcontext = new LdapContextSource();
lcontext.setUrl("${ldap.url}");
lcontext.setBase("${ldap.base}");
lcontext.setUserDn("${ldap.user}");
lcontext.setPassword("${ldap.password}");
return lcontext;
}
#Bean
public LdapTemplate LdapTemplate(){
LdapTemplate lTemplate = new LdapTemplate(ldapContextSource());
return lTemplate;
}
}
In your XML add
<context:annotation-config/>
<bean class="com.mypackage.AppConfig"/>

Connection pool replacement for already implemented Spring Jdbctemplate project

I am a doing a mid size project with spring jdbc and MsSQL server , project is almost 50% done , now when every request doing lots of inserts and updates specially with those tables which contains lots of columns and large datasets is performing very slow , and sometimes showing connection closed.
Now i am thinking to integrate C3p0 or similar connection pooling but i cant change any DAO code which i already done ..
I implemented a DAOHelper class with JDBCTemplate variable and injecting the JDBCTemplate dependency in applicationContext.xml with autowiring of DAOClass in controller class , and i extended this DAOHelper to all DAO classes and using this jdbcTemplate to do JDBC operations.
<bean id="ds" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
<property name="url" value="jdbc:sqlserver://192.168.1.101:1433;databaseName=OrderManager"/>
<property name="username" value="sa"/>
<property name="password" value="520759"/>
</bean>
<bean id="JdbcDataSource" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="ds"/>
</bean>
<bean id="OrderDAO" class="com.ordermanager.order.dao.OrderDAO" >
<property name="jdbcTemplate" ref="JdbcDataSource"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
#Controller
public class OrderController {
#Autowired
OrderDAO orderDAO;
#RequestMapping(value = "/addNewItem", method = RequestMethod.GET)
public ModelAndView addItem(#RequestParam("ParamData") JSONObject paramJson) {
ApplicationContext ctx = new FileSystemXmlApplicationContext(ConstantContainer.Application_Context_File_Path);
OrderDAO orderDAO = (OrderDAO) ctx.getBean("OrderDAO");
return new ModelAndView("MakeResponse", "responseValue", orderDAO.addItem(paramJson));
}
public class DAOHelper {
private JdbcTemplate jdbcTemplate;
private PlatformTransactionManager transactionManager;
public PlatformTransactionManager getTransactionManager() {
return transactionManager;
}
public void setTransactionManager(PlatformTransactionManager txManager) {
this.transactionManager = txManager;
}
public JdbcTemplate getJdbcTemplate() /*I am using this Method for all JDBC Task*/ {
return jdbcTemplate;
}
public void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
Now with minimal code changes how can i integrate C3p0 or any good connection pooling library with my already written code.
Just change the ds bean in your config xml with following and consider adding other c3p0 properties according to your own. make sure to have c3p0 jar in your classpath.
<bean id="ds" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
<property name="jdbcUrl" value="jdbc:sqlserver://192.168.1.101:1433;databaseName=OrderManager" />
<property name="user" value="sa" />
<property name="password" value="520789" />
</bean>

Spring Jms #JmsListener annotation doesnt work

I have such method
#JmsListener(containerFactory = "jmsListenerContainerFactory", destination = "myQName")
public void rceive(MySerializableObject message) {
log.info("received: {}", message);
}
and such config on xml
<jms:annotation-driven />
<bean id="jmsListenerContainerFactory" class="org.springframework.jms.config.DefaultJmsListenerContainerFactory">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="concurrency" value="3-10" />
</bean>
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${brokerURL}" />
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="5" />
<property name="maximumActiveSessionPerConnection" value="500" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="jmsContainerFactory" class="org.springframework.jms.config.DefaultJmsListenerContainerFactory">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="concurrency" value="3-10" />
</bean>
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate" p:connectionFactory-ref="pooledConnectionFactory" />
Seems consumer was not created. I can send messages but cannot receive them.
Any idea whats wrong here?
Just tested your config and it works well. Only difference that I make a class with #JmsListener as a <bean> in that context:
<bean class="org.springframework.integration.jms.JmsListenerAnnotationTests$TestService"/>
#ContextConfiguration
#RunWith(SpringJUnit4ClassRunner.class)
#DirtiesContext
public class JmsListenerAnnotationTests {
#Autowired
private JmsTemplate jmsTemplate;
#Autowired
private TestService testService;
#Test
public void test() throws InterruptedException {
this.jmsTemplate.convertAndSend("myQName", "foo");
assertTrue(this.testService.receiveLatch.await(10, TimeUnit.SECONDS));
assertNotNull(this.testService.received);
assertEquals("foo", this.testService.received);
}
public static class TestService {
private CountDownLatch receiveLatch = new CountDownLatch(1);
private Object received;
#JmsListener(containerFactory = "jmsListenerContainerFactory", destination = "myQName")
public void receive(String message) {
this.received = message;
this.receiveLatch.countDown();
}
}
}
<jms:annotation-driven /> makes the #JmsListener infrastructure available, but to force Spring to see those methods your classes should be beans anyway.
For example <component-scan> for the package with #Service classes.
Cheers!

spring batch - how to execute purge tasklet

I have written a simple spring batch tasklet which calls a dao method which in turn does some deletes. But I am not sure what I should be doing to call the job.
public class RemoveSpringBatchHistoryTasklet implements Tasklet {
#Autowired
private SpringBatchDao springBatchDao;
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
contribution.incrementWriteCount(springBatchDao.purge());
return RepeatStatus.FINISHED;
}
}
So far to execute my spring batch jobs I am using quartz triggers with a setup like so. Each job has it's own xml file which has a read and a writer.
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="jobDetails">
<list>
<ref bean="dailyTranCountJobDetail" />
</list>
</property>
<property name="triggers">
<list>
<ref bean="dailyTranCountCronTrigger" />
</list>
</property>
</bean>
<bean id="dailyTranCountCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="dailyTranCountJobDetail" />
<property name="cronExpression" value="#{batchProps['cron.dailyTranCounts']}" />
</bean>
<bean id="dailyTranCountJobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="com.myer.reporting.batch.JobLauncherDetails" />
<property name="group" value="quartz-batch" />
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="job-daily-tran-counts" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
</map>
</property>
</bean>
And then here is an example of the job file itself with a reader and a writer.
<job id="job-daily-tran-counts" xmlns="http://www.springframework.org/schema/batch">
<step id="job-daily-tran-counts-step1">
<tasklet transaction-manager="custDbTransactionManager">
<chunk
reader="dailyTranCountJdbcCursorItemReader"
writer="dailyTranCountItemWriter"
commit-interval="1000" />
</tasklet>
</step>
</job>
<bean id="dailyTranCountJdbcCursorItemReader"
class="com.myer.reporting.dao.itemreader.DailyTranCountJdbcCursorItemReader"
scope="step"
parent="abstractEposJdbcDao">
<property name="rowMapper">
<bean class="com.myer.reporting.dao.mapper.DailyTranCountMapper" />
</property>
</bean>
<bean id="dailyTranCountItemWriter"
class="com.myer.reporting.dao.itemwriter.DailyTranCountItemWriter"
parent="abstractCustDbJdbcDao"/>
Obviously for this new job there is no reader or writer. So what it he best/correct way for me to execute my new tasklet?
thanks
I prefer java configuration instead of xml. You can configure your tasklet with the following code:
#Configuration
#EnableBatchProcessing
public class BatchCleanUpJobsConfiguration {
#Bean
public Job batchCleanUpJob(final JobBuilderFactory jobBuilderFactory,
final StepBuilderFactory stepBuilderFactory,
final RemoveSpringBatchHistoryTasklet removeSpringBatchHistoryTasklet) {
return jobBuilderFactory.get("batchCleanUpJob")
.start(stepBuilderFactory.get("batchCleanUpStep")
.tasklet(removeSpringBatchHistoryTasklet)
.build())
.build();
}
#Bean
public RemoveSpringBatchHistoryTasklet batchCleanUpTasklet(final JdbcTemplate jdbcTemplate) {
final var tasklet = new RemoveSpringBatchHistoryTasklet();
tasklet.setJdbcTemplate(jdbcTemplate);
return tasklet;
}
}
To schedule your new job use the following code:
#Component
#RequiredArgsConstructor
public class BatchCleanUpJobsScheduler {
private final Job batchCleanUpJob;
private final JobLauncher launcher;
#Scheduled(cron = "0 0 0 * * MON-FRI")
public void launchBatchCleanupJob()
throws JobParametersInvalidException, JobExecutionAlreadyRunningException,
JobRestartException, JobInstanceAlreadyCompleteException {
launcher.run(
batchCleanUpJob,
new JobParametersBuilder()
.addLong("launchTime", System.currentTimeMillis())
.toJobParameters());
}
}

How to inject a dependency bean to GridCacheStore implementation?

My config:
<bean parent="cache-template">
<property name="name" value="yagoLabel" />
<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="distributionMode" value="PARTITIONED_ONLY" />
<property name="backups" value="1" />
<property name="store">
<bean class="id.ac.itb.ee.lskk.lumen.yago.YagoLabelCacheStore" autowire="byType" init-method="init" />
</property>
<property name="writeBehindEnabled" value="true" />
<property name="writeBehindFlushSize" value="102380" />
<property name="writeBehindFlushFrequency" value="30000" />
<property name="writeBehindBatchSize" value="10240" />
<property name="swapEnabled" value="false" />
<property name="evictionPolicy">
<bean class="org.gridgain.grid.cache.eviction.lru.GridCacheLruEvictionPolicy">
<property name="maxSize" value="102400" />
</bean>
</property>
</bean>
And I start GridGain as follows:
My GridCacheStore implementation:
public class YagoLabelCacheStore extends GridCacheStoreAdapter<String, YagoLabel> {
private static final Logger log = LoggerFactory
.getLogger(YagoLabelCacheStore.class);
private DBCollection labelColl;
#GridSpringResource(resourceName="mongoDb")
private DB db;
#Inject
private GridGainSpring grid;
#PostConstruct
public void init() {
log.info("Grid is {}", grid);
labelColl = db.getCollection("label");
}
I start GridGain as follows:
String entityId = "Muhammad";
try (AnnotationConfigApplicationContext appCtx
= new AnnotationConfigApplicationContext(LumenConfig.class)) {
Grid grid = appCtx.getBean(Grid.class);
GridCache<String, YagoLabel> labelCache = YagoLabel.cache(grid);
log.info("Label for {}: {}", entityId, labelCache.get(entityId));
}
LumenConfig Spring configuration contains a DB bean named mongoDb.
However this throws NullPointerException because db is not injected properly. I tried #Inject GridGainSpring just for testing, and even GridGainSpring itself is not injected.
I also tried setting <property name="db" ref="mongoDb"/> in the GridGain Config XML but Spring complains cannot find the bean.
My workaround is to put it inside a public static field but that's soo hacky: https://github.com/ceefour/lumen-kb/blob/b8445fbebd227fb7ac337c758a60badb7ecd3095/cli/src/main/java/id/ac/itb/ee/lskk/lumen/yago/YagoLabelCacheStore.java
The way is to load the GridConfiguration using Spring, then pass it to GridGainSpring.start() :
// "classpath:" is required, otherwise it won't be found in a WAR
#ImportResource("classpath:id/ac/itb/ee/lskk/lumen/core/lumen.gridgain.xml")
#Configuration
public static class GridGainConfig {
#Inject
private ApplicationContext appCtx;
#Inject
private GridConfiguration gridCfg;
#Bean(destroyMethod="close")
public Grid grid() throws GridException {
return GridGainSpring.start(gridCfg, appCtx);
}
}
:-)

Resources