I have this specifically file:
H;COD;CREATION_DATE;TOT_POR;TYPE
H;001;2013-10-30;20;R
D;DETAIL_VALUE;PROP_VALUE
D;003;3030
D;002;3031
D;005;3032
T;NUM_FOL;TOT
T;1;503.45
As you can see, it has header/body/footer lines. I'm looking for a ItemReader that skip these lines. I've done this ItemReader below who identify those lines, using PatternMatchingCompositeLineMapper.
<bean id="fileReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" ref="myFileReference" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.PatternMatchingCompositeLineMapper">
<property name="tokenizers">
<map>
<entry key="H*" value-ref="headerLineTokenizer"/>
<entry key="D*" value-ref="bodyLineTokenizer"/>
<entry key="T*" value-ref="footerLineTokenizer"/>
</map>
</property>
<property name="fieldSetMappers">
<map>
<entry key="H*" value-ref="headerMapper"/>
<entry key="D*" value-ref="bodyMapper"/>
<entry key="T*" value-ref="footerMapper"/>
</map>
</property>
</bean>
</property>
</bean>
I tried to add linesToSkip property equals 1, but it only skipped the header line. Is there a way to skip the first line of each block(header, body and footer)?
Thks.
Nope. linesToSkip (as you wrote) just skip the first linesToSkip lines.
You have to write your own reader using multiorder-line example (or this post) as base and manage skip first line of each block manually
Another option would be this one:
1- Create a Reader Factory
public class CustomFileReaderFactory implements BufferedReaderFactory {
#Override
public BufferedReader create(Resource resource, String encoding) throws IOException {
return new CustomFileReader(new InputStreamReader(resource.getInputStream(), encoding));
}
2- Create your CustomFileReader (this will read one line and decide if we continue or we skip) and make sure to overwrite the readLine() method.
public class CustomFileReader extends BufferedReader {
public CustomFileReader(Reader in) {
super(in);
}
#Override
public String readLine() throws IOException {
String line = super.readLine();
// your logic here
if (hasToBeIgnored(line)) {
return null;
}
return line;
}
3- Set your brand new Factory into your FlatFileItemReader:
yourFlatFileItemReader.setBufferedReaderFactory(new CustomFileReaderFactory());
Related
Use Case :
I would like to launch a job which takes employee id as job parameters, which will be multiple employee ids.
In a file system, files will be residing which contains employee ids as part of the file name (It is a remote file system, not local)
i need to process those files where file name contains the employee-id and passing it to the reader.
I am thinking of using MultiResourceItemReader but i am confused how to match the file name with Employee Id (Job Parameter) which is there in a file system.
Please suggest.
The class MultiResourceItemReader has a method setResources(Resources[] resources) which lets you specify resources to read either with an explicit list or with a wildcard expression (or both).
Example (explicit list) :
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources">
<list>
<value>file:C:/myFiles/employee-1.csv</value>
<value>file:C:/myFiles/employee-2.csv</value>
</list>
</property>
</bean>
Example (wildcard) :
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/myFiles/employee-*.csv" />
</bean>
As you may know, you can use job parameters in configuration by using #{jobParameters['key']} :
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/myFiles/employee-#{jobParameters['id']}.csv" />
</bean>
Unfortunately, wildcard expressions can't manage an OR expression over a list of value with a separator (id1, id2, id3...). And I'm guessing you don't know how many distinct values you'll have to declare an explicit list with a predefined number of variables.
However a working solution would be to use the Loop mechanism of Spring Batch with a classic FlatFileItemReader. The principle is basically to set the next="" on the last step to the first step until you have exhausted every item to read. I will provide code samples if needed.
EDIT
Let's say you have a single chunk to read one file at a time. First of all, you'd need to put the current id from the job parameter in the context to pass it to the reader.
public class ParametersManagerTasklet implements Tasklet, StepExecutionListener {
private Integer count = 0;
private Boolean repeat = true;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// Get job parameter and split
String[] ids = chunkContext.getStepContext().getJobParameters().getString(PARAMETER_KEY).split(DELIMITER);
// Check for end of list
if (count >= ids.length) {
// Stop loop
repeat = false;
} else {
// Save current id and increment counter
chunkContext.getStepContext().getJobExecutionContext().put(CURRENT_ID_KEY, ids[count++];
}
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
if (!repeat) {
return new ExitStatus("FINISHED");
} else {
return new ExitStatus("CONTINUE");
}
}
}
Now you declare this step in your XML and create a loop :
<batch:step id="ParametersStep">
<batch:tasklet>
<bean class="xx.xx.xx.ParametersManagerTasklet" />
</batch:tasklet>
<batch:next on="CONTINUE" to="ReadStep" />
<batch:end on="FINISHED" />
</batch:step>
<batch:step id="ReadStep">
<batch:tasklet>
<batch:chunk commit-interval="10">
<batch:reader>
<bean class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/myFiles/employee-#{jobExecutionContext[CURRENT_ID_KEY]}.csv" />
</bean>
</batch:reader>
<batch:writer>
</batch:writer>
</batch:chunk>
</batch:tasklet>
<batch:next on="*" to="ParametersStep" />
</batch:step>
You can write your own FactoryBean to perform a custom resources search.
public class ResourcesFactoryBean extends AbstractFactoryBean<Resource[]> {
String[] ids;
String path;
public void setIds(String[] ids) {
this.ids = ids;
}
public void setPath(String path) {
this.path = path;
}
#Override
protected Resource[] createInstance() throws Exception {
final List<Resource> l = new ArrayList<Resource>();
final PathMatchingResourcePatternResolver x = new PathMatchingResourcePatternResolver();
for(final String id : ids)
{
final String p = String.format(path, id);
l.addAll(Arrays.asList(x.getResources(p)));
}
return l.toArray(new Resource[l.size()]);
}
#Override
public Class<?> getObjectType() {
return Resource[].class;
}
}
---
<bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
<property name="delegate" ref="itemReader" />
<property name="resources">
<bean class="ResourcesFactoryBean">
<property name="path"><value>file:C:/myFiles/employee-%s.cvs</value> </property>
<property name="ids">
<value>#{jobParameters['id']}</value>
</property>
</bean>
</property>
</bean>
jobParameter 'id' is a comma separated list of your ID.
I have a custom date binder in my controller that uses the following code:
#InitBinder
private void dateBinder(WebDataBinder binder) {
// The date format to parse or output your dates
SimpleDateFormat dateFormat = new SimpleDateFormat("dd/MM/yyyy");
// Create a new CustomDateEditor
CustomDateEditor editor = new CustomDateEditor(dateFormat, true);
// Register it as custom editor for the Date type
binder.registerCustomEditor(Date.class, editor);
}
This is working fine, however I'm trying to apply this globally, to do that, I created a class
public class ClinicBindingInitializer implements WebBindingInitializer {
#Override
public void initBinder(WebDataBinder binder, WebRequest arg1) {
SimpleDateFormat dateFormat = new SimpleDateFormat("dd/MM/yyyy");
// Create a new CustomDateEditor
CustomDateEditor editor = new CustomDateEditor(dateFormat, true);
binder.registerCustomEditor(Date.class, editor);
}
}
and adding the following configuration in my servlet-context.xml
<beans:bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<beans:property name="cacheSeconds" value="0" />
<beans:property name="webBindingInitializer" ref="webBindingInitializer"/>
</beans:bean>
<beans:bean id="webBindingInitializer"
class="br.com.appus.hub.infrastructure.web.ClinicBindingInitializer">
</beans:bean>
and removing the init bind in the controller. However it doesn't work. I checked some places and it seems like it is correct. Is there anything else that I should do?
Edit: Forgot to mention the version of Spring. 3.1.1
Seems like the default AnnotationMethodHandlerAdapter (registered by the annotation-config tag) is handling the request first.
Try setting a order in your handler adapter or use a ConversionService instead.
<beans:bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<beans:property name="cacheSeconds" value="0" />
<beans:property name="webBindingInitializer" ref="webBindingInitializer"/>
<beans:property name="order" value="0" /> <!-- default value is Integer.MAX_VALUE -->
</beans:bean>
I have two classes ClientLogic1 and WelcomeBean1 as follows
public class ClientLogic1 {
public static void main(String[] args)
{
Resource res = new ClassPathResource("spconfig.xml");
BeanFactory factory = new XmlBeanFactory(res);
Object o = factory.getBean("id1");
WelcomeBean1 wb = (WelcomeBean1)o;
wb.show();
}
}
2nd class
public class WelcomeBean1 {
private Map data;
public void setData(Map data) {
this.data = data;
}
public void show()
{
Set s=data.entrySet();
Iterator it = s.iterator();
while(it.hasNext())
{
Map.Entry me = (Map.Entry)it.next();
System.out.println(me.getKey()+ " - "+me.getValue());
}
}
}
I have a xml file as
<beans>
<bean id="id1" class="WelcomeBean1">
<property name="data">
<map>
<entry key="k1">
<vlaue>1323</value>
</entry>
<entry key="k2">
<value>feed</value>
</entry>
</map>
</property>
</bean>
</bean>
I have given the right path.It's just when i run this program i get the following error
Exception in thread "main" java.lang.ClassCastException: WelcomeBean cannot be
cast to mapexmpl.WelcomeBean1 at mapexmpl.ClientLogic1.main(ClientLogic1.java:15)
I am not sure where i am going wrong.Can someone help me plz...
make sure there is no duplicate bean id in spring configuration file. for instance you might have bean WelcomeBean with id id1
change to full package name <bean id="id1" class="mapexmpl.WelcomeBean1">
Actually it is saying as WelcomeBean cannot be ..............But your code is showing all as WelcomeBean1.
You used WelcomeBean some where .Please check it once.
I think before you used WelComeBean.Then changed it to WelComeBean1.Please build agian with clean.
Is there anything that can achieve the equivalent of the below:
<import resource="a.xml">
<prop name="key" value="a"/>
</import>
<import resource="a.xml">
<prop name="key" value="b"/>
</import>
Such that the beans defined in resouce a would see the property key with two different values? The intention would be that this would be used to name the beans in the imports such that resource a.xml would appear:
<bean id="${key}"/>
And hence the application would have two beans named a and b now available with the same definition but as distinct instances. I know about prototype scope; it is not intended for this reason, there will be many objects created with interdepednencies that are not actually prototypes. Currently I am simply copying a.xml, creating b.xml and renaming all the beans using the equivalent of a sed command. I feel there must be a better way.
I suppose that PropertyPlaceholderConfigurers work on a per container basis, so you can't achieve this with xml imports.
Re The application would have two beans named a and b now available with the same definition but as distinct instances
I think you should consider creating additional application contexts(ClassPathXmlApplicationContext for example) manually, using your current application context as the parent application context.
So your many objects created with interdependencies sets will reside in its own container each.
However, in this case you will not be able to reference b-beans from a-container.
update you can postprocess the bean definitions(add new ones) manually by registering a BeanDefinitionRegistryPostProcessor specialized bean, but this solution also does not seem to be easy.
OK, here's my rough attempt to import xml file manually:
disclaimer: I'm very bad java io programmer actually so double check the resource related code :-)
public class CustomXmlImporter implements BeanDefinitionRegistryPostProcessor {
#Override
public void postProcessBeanFactory(
ConfigurableListableBeanFactory beanFactory) throws BeansException {
}
private Map<String, String> properties;
public void setProperties(Map<String, String> properties) {
this.properties = properties;
}
public Map<String, String> getProperties() {
return properties;
}
private void readXml(XmlBeanDefinitionReader reader) {
InputStream inputStream;
try {
inputStream = new ClassPathResource(this.classpathXmlLocation).getInputStream();
} catch (IOException e1) {
throw new AssertionError();
}
try {
Scanner sc = new Scanner(inputStream);
try {
sc.useDelimiter("\\A");
if (!sc.hasNext())
throw new AssertionError();
String entireXml = sc.next();
PropertyPlaceholderHelper helper = new PropertyPlaceholderHelper("${",
"}", null, false);
Properties props = new Properties();
props.putAll(this.properties);
String newXml = helper.replacePlaceholders(entireXml, props);
reader.loadBeanDefinitions(new ByteArrayResource(newXml.getBytes()));
} finally {
sc.close();
}
} finally {
try {
inputStream.close();
} catch (IOException e) {
throw new AssertionError();
}
}
}
private String classpathXmlLocation;
public void setClassPathXmlLocation(String classpathXmlLocation) {
this.classpathXmlLocation = classpathXmlLocation;
}
public String getClassPathXmlLocation() {
return this.classpathXmlLocation;
}
#Override
public void postProcessBeanDefinitionRegistry(
BeanDefinitionRegistry registry) throws BeansException {
XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(registry);
readXml(reader);
}
}
XML configuration:
<bean class="CustomXmlImporter">
<property name="classPathXmlLocation" value="a.xml" />
<property name="properties">
<map>
<entry key="key" value="a" />
</map>
</property>
</bean>
<bean class="CustomXmlImporter">
<property name="classPathXmlLocation" value="a.xml" />
<property name="properties">
<map>
<entry key="key" value="b" />
</map>
</property>
</bean>
this code loads the resources from classpath. I would think twice before doing something like that, anyway, you can use this as a starting point.
I am having a requirement wherein I need to read the contents of a text file through spring framework. For this purpose I made a method in my service implementation class as below-
public String readFile(File file)
This method will take the file name as input and read the file.
I was writing the code in XML for spring as below-
<bean id="fstream" class="java.io.FileInputStream">
<constructor-arg value="C:/text.txt" />
</bean>
<bean id="in" class="java.io.DataInputStream">
<constructor-arg ref="fstream"/>
</bean>
<bean id="isr" class="java.io.InputStreamReader">
<constructor-arg ref="in"/>
</bean>
<bean id="br" class="java.io.BufferedReader">
<constructor-arg ref="isr"/>
</bean>
Following code goes in my method-
public String readFile(File file)
{
String line = null;
String content = "";
try
{
ApplicationContext context = new ClassPathXmlApplicationContext("FileDBJob.xml");
BufferedReader br = (BufferedReader) context.getBean("br");
while((line = br.readLine())!=null)
content = content.concat(line);
}
catch (Exception e)
{
e.printStackTrace();
}
return content;
}
But here the issue is that i need to hardcode the file name in XML, so there is no use of file parameter.
Kindly help in finding the solution. As I am new to spring and trying my hands with it so it may be possible that I am missing something. Any help would be of great help.
Don't inject the streams and readers, that's not really how Spring is intended to be used. I'd inject the file itself:
public class MyFileReader {
private File file;
public String readFile() {
StringBuilder builder = new StringBuilder();
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(getFile()));
String line = null;
while ((line = reader.readLine()) != null)
builder.append(line);
} catch (IOException e) {
e.printStackTrace();
} finally {
closeQuietly(reader);
}
return builder.toString();
}
private void closeQuietly(Closeable c) {
if (c != null) {
try {
c.close();
} catch (IOException ignored) {}
}
}
public File getFile() {
return file;
}
public void setFile(File file) {
this.file = file;
}
}
Then your bean def looks like this:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:app.properties"/>
</bean>
<bean class="com.myapp.MyFileReader">
<property name="file" value="${filePath}" />
</bean>
All that is left is to create your app.properties file with the correct info. You can also set the value by invoking the app with a -DfilePath=/foo/bar/whatever.txt
I have tested this code its working.....
Try to implement....you have to copy paste schedular.xml file in ur proj configuration folder(where applicationContext.xml file in ur application and it has to be
contextConfigLocation
WEB-INF/config/*.xml
in ur web.xml file).
Then configure SvhedularTask bean in ur service classes xml file....it will trigger for every minute.
////SCHEDULARTASK.JAVA//////
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Date;
import javax.servlet.ServletContext;
import org.springframework.web.context.ServletContextAware;
/**
* The Class SchedulerTask.
*/
public class SchedulerTask implements ServletContextAware{
private ServletContext servletContext;
#Override
public void setServletContext(ServletContext arg0) {
// TODO Auto-generated method stub
this.servletContext = arg0;
}
public void unZipProcess() throws IOException{
System.out.println(servletContext);
File folder = new File("C:/Users/rerrabelli/Desktop/test");
File[] listOfFiles = folder.listFiles();
if (listOfFiles != null){
for (int i = 0; i < listOfFiles.length; i++) {
if (listOfFiles[i].isFile()) {
if (listOfFiles[i].getName().endsWith(".txt")) {
File file = new File("C:/Users/rerrabelli/Desktop/test" + File.separator
+ listOfFiles[i].getName());
long millisec = file.lastModified();
Date dt = new Date(millisec);
long difference = new Date().getTime()-dt.getTime();
System.out.println((difference/1000)/60);
if(((difference/1000)/60)<1){
FileInputStream fin = new FileInputStream(
file);
ByteArrayOutputStream tmp = new ByteArrayOutputStream();
byte b;
while ((b = (byte) fin.read()) != -1) {
tmp.write(b);
}
byte[] customerData = tmp.toByteArray();
String data = new String(customerData);
System.out.println(data);
servletContext.setAttribute(file.getName(), data);
}
}
}
}
}
System.out.println(servletContext.getAttribute("test.txt"));
}
}
//////APPLICATION CONTEXT.xml/////////
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">
<bean id="schedulerTask" class="com.altimetrik.simreg.service.impl.SchedulerTask">
</bean>
</beans>
======================
SCHEDULAR.XML
===========
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN/EN" "http://www.springframework.org/dtd/spring-beans.dtd" >
<beans>
<import resource="applicationContext.xml"/>
<bean id="schedulerTask1"
class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
<property name="targetObject"> <ref bean="schedulerTask" /> </property>
<property name="targetMethod"> <value>unZipProcess</value> </property>
<property name="concurrent"> <value>false</value> </property>
</bean>
<bean id="UnzipTrigger"
class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail"> <ref bean="schedulerTask1" /> </property>
<property name="cronExpression"> <value>0 0/1 * * * ?</value> </property>
</bean>
<bean
class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<list>
<!-- Add triggers here-->
<ref bean="UnzipTrigger" />
</list>
</property>
</bean>
</beans>