I am using Spring Batch 2. version.I have generated the csv file and able to save in csv format on local.
Now I want to generate the same file but it will be stored on SFTP server.
I had gone through some tutorial which generates file on sftp server but they are using spring integration with Spring Batch.
Is it possible to generate the file on SFTP server using Spring Batch only?
Below is itemReader bean defined::
<bean id="itemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="select u.ID, u.USER_LOGIN, u.PASSWORD, u.AGE from USERS u" />
</property>
<property name="rowMapper">
<bean class="com.example.UserRowMapper" />
</property>
</bean>
ItemWriter Bean::
<bean id="flatFileItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:csv/user.csv" />
<property name="appendAllowed" value="true" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="," />
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="name,age,id,password"/>
</bean>
</property>
</bean>
You have to customize your writer class
Related
I need to create a batch job that reads from the database millions of rows and I am thinking which is the best approach to do it the fastest that can be done; I think that using sql native to read will be the best option but I am also want to know more infos about the spring batch configuration and I don't seem to finding what I am searching for.
I know there are 2 ways of reading data in a Spring batch job using a Reader.
inside a bean eg:
<batch:step id="fetchData" >
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="dataReader" processor="dataProcessor"
writer="dataWriter" commit-interval="100"/>
</batch:tasklet>
<bean id="dataReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSourceNative"/>
<property name="pageSize" value="100"/>
<property name="rowMapper" ref="RowsMapper"/>
<property name="queryProvider">
<bean class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSourceNative"/>
<property name="selectClause"
value="SELECT a,b "/>
<property name="fromClause"
value="from table"/>
<property name="whereClause"
value="WHERE a="blabla"/>
</bean>
</property>
</bean>
using a Java Class
<property name="targetObject" ref="myClassDao" />
<property name="targetMethod" value="list" />
<property name="arguments">
<list>
<!-- add arguments list -->
</list>
</property>
</bean>
<bean id="myClassDao" class="path.to.MyClassDAO"
Is it the same thing to do any of this approaches or the 2nd one gives you more liberty?
If the answer is no, which option is better/fastest?
I am thinking if I want to read from the database but using hibernate to do it. Can be implemented with any of these approaches?
Currently I am using JdbcCursorItemReader and FlatFileItemWriter in a job step.
Due to performance issue we have to use stored procedure.
Is there a way to make a call to stored proc in Spring Batch 2.0.. RELEASE?
<bean id="jdbcCursorItemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource"/>
<property name="sql"
value="SELECT X,Y,Z
FROM V_VIEW "/>
<property name="mapper">
<bean class="com.mapping.SomeMapper"/>
</property>
</bean>
<bean class="org.springframework.batch.item.file.FlatFil eItemWriter" id="flatFileItemWriter">
<property name="resource" ref="resource"/>
<property name="fieldSetCreator">
<bean class="org.springframework.batch.item.file.mapping .PassThroughFieldSetMapper"/>
</property>
</bean>
... Other config
how to write a custom database reader wih callable statement...
Sample code is appreciated.. Thank You,.
There is a StoredProceedureItemReader that is built just for this use case. You can read more about it in the documentation here: https://docs.spring.io/spring-batch/apidocs/org/springframework/batch/item/database/StoredProcedureItemReader.html
I could find my answer:
<bean id="jdbcCursorItemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource"/>
<property name="sql" value="Call schema.StoredProcName"/>
<property name="mapper">
<bean class="com.mapping.SomeMapper"/>
</property>
</bean>
I am new to spring and my java is rusty so please excuse me if this is a simple question.
I have a two step job. The first step is to query a MongoDB and write a file. The second step is to read the file and process it.
I am using FlatFileItemReader for the second step. In the first step, I am writing the file without specifying any path.
When I run the job from within sts, the first step creates the file in the project directory in my sts workspace folder.
The second step throws the input resource must exist exception. I tried a variety of options such as file://, file:/ and so on without any success.
Confused as to why the read doesn't look for the file in the same place that the write wrote to.
In any case, any help in resolving the issue will be appreciated.
The first step does not use a FlatFileItemWriter and is just a tasklet that writes the file using java.
Here's the config:
<batch:step id="step2">
<batch:tasklet>
<batch:chunk reader="customerItemReader" processor="customerProcessor" writer="mongodbItemWriter"
commit-interval="1">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="loadFromMongo" class="org.springframework.batch.core.step.tasklet.MethodInvokingTaskletAdapter">
<property name="targetObject">
<bean class="testMongo.MongoLoader"/>
</property>
<property name="targetMethod" value="loadFromMongo" />
</bean>
<bean id="customerItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="classpath:Customers.txt"/>
<property name="lineMapper" ref="customerLineMapper"/>
<bean id="customerLineMapper"
class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="customerLineTokenizer"/>
<property name="fieldSetMapper" ref="customerFieldSetMapper"/>
<bean id="customerLineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
</bean>
<bean id="customerFieldSetMapper" class="testMongo.CustomerFieldSetMapper">
</bean>
<bean id="customerProcessor" class="testMongo.CustomerItemProcessor">
</bean>
<bean id="mongodbItemWriter" class="org.springframework.batch.item.data.MongoItemWriter">
<property name="template" ref="mongoTemplate" />
<property name="collection" value="creditReport" />
</bean>
<!-- commenting out copied stuff from mkyong SpringBatch Example for now
<bean id="xmlItemReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="record" />
<property name="resource" value="classpath:xml/report.xml" />
<property name="unmarshaller" ref="reportUnmarshaller" />
</bean>
<bean id="reportUnmarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases">
<util:map id="aliases">
<entry key="record" value="com.mkyong.model.Report" />
</util:map>
</property>
</bean>
-->
I am trying to create an SSL connection to activemq (using Spring 3.1, ActiveMQ 5.5, and Camel 10.0). I'm getting the dreaded SSL handshake exception. I can connect with an openssl s_client using the certificate in the jks. Thus, I'm trying to figure out if there is a problem with my keystore (which seems to work in other situations - e.g. with tomcat) or a problem with my XML configuration. Does anyone have a good example, thoughts about other ways to test, or see what I'm doing wrong?
Please note, my config passes validation (in case you see a typo).
Thanks
-J
<bean id="myJmsRedeliverPolicy" class="org.apache.activemq.RedeliverPolicy">
<property name="maximumRedeliveries" value="500"/>
</bean>
<bean id="jmsSecureConnectionFactory" class="org.apache.activemq.ActiveMQSslConnectionFactory">
<property name="brokerURL" value="ssl://test.com:8100"/>
<property name="redeliverPolicy" ref="myJmsRedeliverPolicy"/>
<property name="keyStore" value="/usr/lib/mykeystore.jks"/>
<property name="keyStorePassword" value="mypass"/>
<property name="trustStore" value="/usr/lib/mycacerts"/>
<property name="trustStorePassword" value="changeit"/>
</bean>
<bean id="pooledSecureConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="10"/>
<property name="connectionFactory" ref="jmsSecureConnectionFactory"/>
</bean>
<bean id="txSecureManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="pooledSecureConnectionFactory" />
</bean>
<bean id="jmsSecureConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledSecureConnectionFactory" />
<property name="testConnectionOnStartup" value="true"/>
<property name="transacted" value="true"/>
<property name="transactionManager" ref="txSecureManager"/>
</bean>
<bean id="activemqs" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsSecureConfig" />
</bean>
Start your application with:
-Djavax.net.debug=ssl
to get further with your troubleshooting. Usually, that command provides printout that says pretty much exactly what's wrong.
I have created a JAR that I need to use in my WEB-APP. Both are created with spring framework. I would like to load a .properties file outside the JAR file, in the main context of the web-application. And I want to do it with the facilities that Spring offers us.
I've tried to do something like this in my spring.xml file inside the JAR:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location">
<value>/WEB-INF/classes/my.properties</value>
</property>
</bean>
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<bean id="cronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="myJob" />
<property name="cronExpression" value="${my.cronExpression}"/>
</bean>
</property>
</bean>
Trying to load my.cronExpression from my.properties file. But without any success.
I always get this error:
Could not resolve placeholder 'my.cronExpression'.
I've tried to change the location with many variants, using classpath:/WEB-INF/classes/my.properties etc...
But I'm not able to load the configuration file.
Thanks for your help.
Use classpath:my.properties - /WEB-INF/classes is root of your classpath.
Try declaring it as follows:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location">
<value>/WEB-INF/classes/my.properties</value>
</property>
<property name="ignoreUnresolvablePlaceholders">
<value>true</value>
</property>
</bean>
I have gone through your code and want you to try this code snippet
It works well for me :)
<bean id="placeholderProperties"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="file:/WEB-INF/classes/my.properties" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="ignoreUnresolvablePlaceholders" value="true" />
<property name="order" value="1" />
</bean>