Reading in UTF-8 encoding in Java results in issues at only some places - utf-8

I am reading a UTF_8 encoded file byte by byte in Java Spring Batch and saving the content in mongodb. The reading is delegated to FlatFileItemReader and the encoding is injected into the bean as UTF-8.
What I do see is that only at one or two specific places, the umlaut character is getting stored with an encoding problem in mongodb. I verified that the content in the original file is okay. What is interesting is that everywhere else, the umlaut character in the original file is interpreted correctly and stored correctly in mongodb.
Could you help with this?
<bean id="recordReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="file:#{jobExecutionContext['inputfile']}" />
<property name="linesToSkip" value = "1" />
<property name= "encoding" value ="UTF-8"/>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper" >
<property name = "lineTokenizer">
<bean class = "com.xyz.NoQuoteDelimitedLineTokenizer">
<property name = "strict" value = "false" />
<property name = "names" value = "Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8" />
<property name="delimiter">
<util:constant static-field="org.springframework.batch.item.file.transform.DelimitedLineTokenizer.DELIMITER_TAB" />
</property>
</bean>
</property>
<property name = "fieldSetMapper">
<bean class = "com.xyz.MyFieldSetMapper" >
<property name = "ctx" value="#{jobExecutionContext['contextBean']}"/>
</bean>
</property>
</bean>
</property>
</bean>
FieldSetMapper
private void validateFields (String id, ConsumerDO ci, FieldSet fieldset_p) throws Exception
{try {
ci.setCaption(fieldset_p.readString("newCaption"));
}
catch (Exception e) {
}
}

Related

Spring Batch - Last item from the reader alone is getting updated

I have to read from a file (FlatFile) and update a column if that ID present in the file matches the id in the column.The file is being read properly but only the last id value is getting updated here . Please find the snippet
Job-Config.xml
<bean id="abcitemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="file:datafile/outputs/ibdData.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="ID,NAM,TYPE" />
<property name="delimiter" value="|"/>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.pershing.intraware.springbatch.mapper.abcFieldsetMapper" />
</property>
</bean>
</property>
</bean>
<bean id="abcitemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"><value>UPDATE TEST_abc SET BIZ_ARNG_CD = CASE WHEN ID IN (SELECT ID FROM TEST_abc WHERE ID= ? and MONTH=(to_char(sysdate, 'MM')) AND YR =(to_char(sysdate, 'YY'))) THEN 'Y' ELSE 'N' END</value></property>
<!-- It will take care matching between object property and sql name parameter -->
<property name="itemPreparedStatementSetter" ref="testPrepStatementSetter" />
</bean>
</beans>
Setter.java
public class IDItemPreparedStatementSetter implements ItemPreparedStatementSetter<Test> {
#Override
public void setValues(Test item, PreparedStatement ps) throws SQLException {
// TODO Auto-generated method stub
ps.setString(1, item.getID());
}
}
Your query is updating each row of database every time it is fired. You need to restrict that. Currently; it must be setting the BIZ_ARNG_CD to 'Y' for records with ID equal to the ID of the last record passed to the writer.
You can fix this in 2 ways -
Default the database column to 'N' and don't set it to 'N' in the update statement
Add where clause in update script ( BIZ_ARNG_CD != 'Y')

read only selective columns from csv file using spring batch

there is one csv file having 100 columns, but we need only 3-5 columns which needs to be loaded into database.
I dont want to specify all the 100 columns in linetokenizer in job xml.
Please suggest how we can proceed in this case
Try using a custom fieldSetMapper. You can use it similar to a ResultSet with indexes.
You have to list all the column names only if you want automatic mapping.
Specify only the delimiter, in your case ","
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="YOURFILE" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<bean class="CUSTOMFIELDSETMAPPER" />
</property>
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="," />
</bean>
</property>
</bean>
</property>
</bean>
The Custom Mapper could be something like this, say if you want to read the 1st column and 25th column:
public class CustomMapper implements FieldSetMapper<CustomPOJO>{
#Override
public CustomPOJO mapFieldSet(FieldSet fieldSet) throws BindException {
CustomPOJO result = new CustomPOJO();
result.setName(fieldSet.readString(0));
result.setAddress(fieldSet.readString(24));
return result;
}
}
For further explanation on how to use the reader, please refer to this tutorial

Spring Controller #ResponseBody text/xml response UTF-8 encoding issue

I have annotation based Spring Rest Service running on jetty web server(also tomcat).The controller code is :
#RequestMapping(method = RequestMethod.POST, value = { "/ssrfeed/exec/",
"/query/exec" }, consumes = { "application/xml", "text/xml",
"application/x-www-form-urlencoded" }, produces = {
"application/xml;charset=UTF-8", "text/xml;charset=UTF-8",
"application/x-www-form-urlencoded;charset=UTF-8" })
#ResponseBody
protected String getXmlFeed(HttpServletRequest request,
#PathVariable String serviceName, #RequestBody String xmlReq) {
//code....
return appXMLResponse;
}
The problem is that the response xml returned by Controller contains some characters like ä ö ü (Umlaute). The response when rendered on browser gives the parsing error :
XML Parsing Error: not well-formed
Location: //localhost:8083/MySerice/ssrfeed/exec/
Line Number 18111, Column 17:
<FIRST_NAME>Tzee rfista</FIRST_NAME>
----------------^
(a small triangle appear in place of ü)
The expected is : <FIRST_NAME>Tzeeürfista</FIRST_NAME>
I have tried a below solutions but issue is still there.
Tried using filters referring to solution given on technowobble
passed the charset to StringHttpMessageConverter property
<bean id="restTemplate" class="org.springframework.web.client.RestTemplate">
<property name="messageConverters">
<list>
<bean class="org.springframework.http.converter.json.MappingJacksonHttpMessageConverter">
<property name="supportedMediaTypes" value="application/json" />
</bean>
<bean class="org.springframework.http.converter.StringHttpMessageConverter">
<property name="supportedMediaTypes" value="text/xml;charset=UTF-8" />
</bean>
</list>
</property>
</bean>
ref link
Enabled the SetCharacterEncodingFilter in tomcat -web.xml
Changed the code to return ResponseEntity instead of String and removed #ResponseBody.
protected ResponseEntity<String> getXmlFeed(HttpServletRequest
request, #PathVariable String serviceName, #RequestBody String xmlReq) {
//line of code
HttpHeaders responseHeaders = new HttpHeaders();
responseHeaders.add("Content-Type", "application/xml; charset=utf-8");
return new ResponseEntity<String>(appXMLResponse, responseHeaders, HttpStatus.CREATED);
}
The 4th solution works But this being existing code I can't change method signature as it might impact existing clients of this service. Any ideas/pointers to solve this ?
in your dispatcher servlet context xml, you have to add a propertie. e.g.
<bean class = "org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<property name="messageConverters">
<array>
<bean class = "org.springframework.http.converter.StringHttpMessageConverter">
<property name="supportedMediaTypes" value = "text/plain;charset=UTF-8" />
</bean>
</array>
</property>
</bean>
Finally the issue is resolved. Here is what I did.
1. Used StringHttpMessageConverter's constructer for setting charset as :
<bean id="stringHttpMessageConverter"
class="org.springframework.http.converter.StringHttpMessageConverter">
<constructor-arg index="0" name="defaultCharset" value="UTF-8"/>
<property name="supportedMediaTypes">
<list>
<value>application/xml</value>
<value>text/xml</value>
<value>application/x-www-form-urlencoded</value>
</list>
</property>
</bean>
Also I removed the unnecessary spring3.0 and 3.1 jars from my project. These were not required but were lying there. (should have done earlier).
This solved the problem for me.
There is no such answer with what I've solved my encoding problem so I'll post it.
I've got Spring RestService running on Jetty. At response body part of data that was received from database had correct UTF-8 encoding, but data from .property file (with error and success messages) had incorrect encoding and was like äöü...
At first I checked encoding of .property file itself with File->Settings->Editor->Code Style-> File Encodings (in such way you could not only check but set encoding you need) - it was UTF-8.
Then I set response encoding #RequestMapping in my RestController:
#RequestMapping(value = "/category/{categoryId}", method = RequestMethod.DELETE, produces = { "application/json;**charset=UTF-8**" })
and set defaultCharset property for Jackson2:
<bean id="jsonConverter" class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter">
<property name="supportedMediaTypes" value="application/json;" />
<property name="prettyPrint" value="true" />
<property name="defaultCharset" value="UTF-8"/>
</bean>
No result.
But then I found that the problem could be solved by adding UTF-8 encoding to
PropertyPlaceholderConfigurer who grabs data from my .property file:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>classpath:app.properties</value>
<value>classpath:database.properties</value>
<value>classpath:ru.error.messages.properties</value>
<value>classpath:ru.success.messages.properties</value>
</list>
</property>
<property name="fileEncoding" value="UTF-8"/>
</bean>
... and the problem has gone )))

Spring 3.2 with MVC, ContentNegotation, REST and PDF Generator

Let's say, I have a REST styled controller mapping
#RequestMapping(value="users", produces = {MediaType.APPLICATION_JSON_VALUE})
public List<User> listUsers(#ReqestParams Integer offset, #ReqestParams Integer limit, #ReqestParams String query) {
return service.loadUsers(query, offset, limit);
}
Serving JSON (or even XML) is not an issue, this is easy using ContentNegotation and MessageConverters
<bean id="contentNegotiationManager" class="org.springframework.web.accept.ContentNegotiationManagerFactoryBean">
<property name="favorPathExtension" value="true" />
<property name="favorParameter" value="false" />
<property name="ignoreAcceptHeader" value="false" />
<property name="mediaTypes" >
<value>
html=text/html
json=application/json
xml=application/xml
</value>
</property>
</bean>
Now, I need to add support for PDF. Naturally, I want to use (Spring) MVC + REST as much as possible. Most examples I have found implement this with an explicit definition not using REST style, e.g.
#RequestMapping(value="users", produces = {"application/pdf"})
public ModelAndView listUsersAsPdf(#ReqestParams Integer offset, #ReqestParams Integer limit, #ReqestParams String query) {
List<User> users = listUsers(offset, limit, query); // delegated
return new ModelAndView("pdfView", users);
}
That works, but is not very comfortable because for every alternate output (PDF, Excel, ...) I would add a request mapping.
I have already added application/pdf to the content negotation resolver; unfortunately any request with a suffix .pdf or the Accept-Header application/pdf were be responded with 406.
What is the ideal setup for a REST/MVC style pattern to integrate alternate output like PDF?
You can create a WEB-INF/spring/pdf-beans.xml like below.
<bean id="listofusers" class="YourPDFBasedView"/>
And your controller method will return view name as listofusers.
#RequestMapping(value="users")
public ModelAndView listUsersAsPdf(#ReqestParams Integer offset, #ReqestParams Integer limit, #ReqestParams String query) {
List<User> users = listUsers(offset, limit, query); // delegated
return new ModelAndView("listofusers", users);
}
And you can use contentNegotiationViewResolver in this way:
<bean class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="order" value="1"/>
<property name="location" value="WEB-INF/spring/pdf-views.xml"/>
</bean>
<!--
View resolver that delegates to other view resolvers based on the content type
-->
<bean class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver">
<!-- All configuration is now done by the manager - since Spring V3.2 -->
<property name="contentNegotiationManager" ref="cnManager"/>
</bean>
<!--
Setup a simple strategy:
1. Only path extension is taken into account, Accept headers are ignored.
2. Return HTML by default when not sure.
-->
<bean id="cnManager" class="org.springframework.web.accept.ContentNegotiationManagerFactoryBean">
<property name="ignoreAcceptHeader" value="true"/>
<property name="defaultContentType" value="text/html" />
</bean>
For JSON: Create a generic JSON view resolver like below and register it as bean in context file.
public class JsonViewResolver implements ViewResolver {
/**
* Get the view to use.
*
* #return Always returns an instance of {#link MappingJacksonJsonView}.
*/
#Override
public View resolveViewName(String viewName, Locale locale) throws Exception {
MappingJacksonJsonView view = new MappingJacksonJsonView();
view.setPrettyPrint(true); // Lay the JSON out to be nicely readable
return view;
}
}
Same for XML:
public class MarshallingXmlViewResolver implements ViewResolver {
private Marshaller marshaller;
#Autowired
public MarshallingXmlViewResolver(Marshaller marshaller) {
this.marshaller = marshaller;
}
/**
* Get the view to use.
*
* #return Always returns an instance of {#link MappingJacksonJsonView}.
*/
#Override
public View resolveViewName(String viewName, Locale locale)
throws Exception {
MarshallingView view = new MarshallingView();
view.setMarshaller(marshaller);
return view;
}
}
and register above xml view resolver in context file like this:
<oxm:jaxb2-marshaller id="marshaller" >
<oxm:class-to-be-bound name="some.package.Account"/>
<oxm:class-to-be-bound name="some.package.Customer"/>
<oxm:class-to-be-bound name="some.package.Transaction"/>
</oxm:jaxb2-marshaller>
<!-- View resolver that returns an XML Marshalling view. -->
<bean class="some.package.MarshallingXmlViewResolver" >
<constructor-arg ref="marshaller"/>
</bean>
You can find more information at this link:
http://spring.io/blog/2013/06/03/content-negotiation-using-views/
Using all view resolver techniques, you can avoid writing duplicate methods in controller, such as one for xml/json, other for excel, other for pdf, another for doc, rss and all.
Knalli, if you replace #ResponseBody with ModelAndView(), you can achieve both the features.
Is there any reason you want to keep #ResponseBody ? I just want to know if I am missing anything, just want to learn.
Other option is to write HttpMessageConverters then:
Some samples are here.
Custom HttpMessageConverter with #ResponseBody to do Json things
http://www.javacodegeeks.com/2013/07/spring-mvc-requestbody-and-responsebody-demystified.html
This is working sample. I have configured contentnegotiationviewresolver for this, and give highest order. After that I have ResourceBundleViewResolver for JSTL and Tiles View, then XmlViewResolver for excelResolver, pdfResolver, rtfResolver. excelResolver, pdfResolver, rtfResolver. XmlViewResolver and ResourceBundleViewResolver works only with MAV only, but MappingJacksonJsonView and MarshallingView takes care for both MAV and #ResponseBody return value.
<bean id="contentNegotiatingResolver" class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver">
<property name="order"
value="#{T(org.springframework.core.Ordered).HIGHEST_PRECEDENCE}" />
<property name="mediaTypes">
<map>
<entry key="json" value="application/json" />
<entry key="xml" value="application/xml" />
<entry key="pdf" value="application/pdf" />
<entry key="xlsx" value="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" />
<entry key="doc" value="application/msword" />
</map>
</property>
<property name="defaultViews">
<list>
<!-- JSON View -->
<bean class="org.springframework.web.servlet.view.json.MappingJacksonJsonView" />
<!-- XML View -->
<bean class="org.springframework.web.servlet.view.xml.MarshallingView">
<constructor-arg>
<bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
<property name="classesToBeBound">
<list>
<value>Employee</value>
<value>EmployeeList</value>
</list>
</property>
</bean>
</constructor-arg>
</bean>
</list>
</property>
<property name="ignoreAcceptHeader" value="true" />
</bean>
<bean class="org.springframework.web.servlet.view.ResourceBundleViewResolver"
id="resourceBundleResolver">
<property name="order" value="#{contentNegotiatingResolver.order+1}" />
</bean>
<bean id="excelResolver" class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="location">
<value>/WEB-INF/tiles/spring-excel-views.xml</value>
</property>
<property name="order" value="#{resourceBundleResolver.order+1}" />
</bean>
<bean id="pdfResolver" class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="location">
<value>/WEB-INF/tiles/spring-pdf-views.xml</value>
</property>
<property name="order" value="#{excelResolver.order+1}" />
</bean>
<bean id="rtfResolver" class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="location">
<value>/WEB-INF/tiles/spring-rtf-views.xml</value>
</property>
<property name="order" value="#{excelResolver.order+1}" />
</bean>
And our XMLViewResolver spring-pdf-views.xml looks like this.
<bean id="employees"
class="EmployeePDFView"/>
And EmployeePDFView will have code for generating pdf and writing pdf byte stream on Response object. This will resolve to rest url that will end with .pdf extension, and when you return MAV with "employees" id.

How can I get the number of line and the filename in itemProcessor - spring batch

I am using spring batch to parse my files. In ItemProcessor I validate if the incoming fields are correct. If they are not I want to throw a ValidationException and log to a file the corresponding row which has the incorrect fields. So, how can I find the number of line and the filename in ItemProcessor?
Without seeing you ItemReader config I can't really be sure but if you are using something like FlatFileItemReader to parse a csv, if in strict mode it will validate the number of columns.
Assuming you reader looks like this, that is:
<bean id="iItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="linesToSkip" value="1"/>
<property name="comments" value="#" />
<property name="encoding" value="UTF-8"/>
<property name="lineMapper" >
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=","/>
<property name="names">
<list >
<value>First_Field</value>
<value>Second_Field</value>
</list>
</property>
<property name="strict" value="true"/>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="uk.co.package.FieldSetMapper">
<property name="dateFormat" value="yyyy-MM-dd HH:mm:ss"/>
</bean>
</property>
</bean>
</property>
</bean>
It will throw a FlatFileParseException for any lines that can't be processed. This includes the line number and can be handled in a listener.
As for the line number, you might build your own LineMapper and then store the line-number in your business object. An example in which I store the line unprocessed (as-is) together with the line number:
DefaultLineMapper<OneRow> lineMapper = new DefaultLineMapper<OneRow>() {
#Override
public OneRow mapLine(String line, int lineNumber) throws Exception {
return new OneRow(lineNumber, line);
}
};
Of course you can already map your Object, I had the need to have the whole line unprocessed as input to my Processors.
As a reference with the same idea: https://stackoverflow.com/a/23770421/5658642

Resources