Spring batch MongoItemReader to read as json - spring

I am trying to write a Spring Batch job that reads documents from a mongo db and writes the documents to a CMS (for now I will attempt to test this first with WireMock). Can I set up the job and itemreader without specifying the exact structure of the document? I would just like to read each document as json and then sent that json through to the CMS. Is this even possible?

Since JSON is just a String, you should configure your MongoItemReader for String type and provide MongoTemplate with some custom simple converter:
public class DBObjectToStringConverter implements Converter<DBObject, String> {
public String convert(DBObject source) {
return source == null ? null : source.toString();
}
}
This one just return a String JSON representation of DBObject.
Then configuration:
<mongo:db-factory/>
<mongo:mapping-converter id="mappingConverter">
<mongo:custom-converters>
<mongo:converter>
<bean class="com.my.batch.mongo.DBObjectToStringConverter "/>
</mongo:converter>
</mongo:custom-converters>
</mongo:mapping-converter>
<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg ref="mongoDbFactory"/>
<constructor-arg ref="mappingConverter"/>
</bean>
<bean class="org.springframework.batch.item.data.MongoItemReader">
<property name="template" ref="mongoTemplate"/>
<property name="query" value="..."/>
<property name="targetType" value="java.lang.String"/>
</bean>
And voila! Each item returns as JSON String.

Related

read only selective columns from csv file using spring batch

there is one csv file having 100 columns, but we need only 3-5 columns which needs to be loaded into database.
I dont want to specify all the 100 columns in linetokenizer in job xml.
Please suggest how we can proceed in this case
Try using a custom fieldSetMapper. You can use it similar to a ResultSet with indexes.
You have to list all the column names only if you want automatic mapping.
Specify only the delimiter, in your case ","
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="YOURFILE" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<bean class="CUSTOMFIELDSETMAPPER" />
</property>
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="," />
</bean>
</property>
</bean>
</property>
</bean>
The Custom Mapper could be something like this, say if you want to read the 1st column and 25th column:
public class CustomMapper implements FieldSetMapper<CustomPOJO>{
#Override
public CustomPOJO mapFieldSet(FieldSet fieldSet) throws BindException {
CustomPOJO result = new CustomPOJO();
result.setName(fieldSet.readString(0));
result.setAddress(fieldSet.readString(24));
return result;
}
}
For further explanation on how to use the reader, please refer to this tutorial

indexing document to a specific collection using spring data solr

I am trying to index a document to a specific collection in solr. The collection name is 'program'. I am using spring data solr.
I am getting the below error when trying to save the document:
HTTP ERROR 404
Problem accessing /solr/update.
Reason:Not Found
My assumption is that the annotation #SolrDocument is not recognized. spring-data-solr is trying to post the document to /solr/update whereas it should try to post it to /solr/program/update.However I am not sure how to prove it or fix it.
My schema is available on the link below:
http://<solr-host>/solr/program/schema
The update request handler is available in the link below:
http://<solr-host>/solr/program/update
spring-config-xml
<context:annotation-config />
<context:component-scan base-package="com.oostnet.controllers, com.oostnet.models, com.oostnet.services" />
<mvc:annotation-driven />
<solr:solr-server id="solrServer" url="http://<solr-host>/solr" />
<bean id="solrTemplate" class="org.springframework.data.solr.core.SolrTemplate">
<constructor-arg ref="solrServer" />
</bean>
<solr:repositories base-package="com.oostnet.solr.repositories" />
Model definition:
package com.oostnet.models.documents;
#SolrDocument(solrCoreName="program")
public class Program {
#Id
#Indexed
private String id;
<more variables>
}
Repository definition:
package com.oostnet.solr.repositories;
public interface ProgramRepository extends SolrCrudRepository<Program, String> {
}
Controller:
package com.oostnet.controllers;
public class ProgramController{
private ProgramRepository programRepository;
#Autowired
public void setProgramRepository(ProgramRepository programRepository) {
this.programRepository = programRepository;
}
public void createProgram(Program program) {
programRepository.save(program);
}
}
Below are the versions used:
<spring.data.solr.version>1.3.2.RELEASE</spring.data.solr.version>
<spring.version>4.1.4.RELEASE</spring.version>
solr server version - solr-spec 4.10.2
To work on multiple collection you need to enable multiple repository under spring configuration as below :-
<context:annotation-config />
<context:component-scan base-package="com.test.solr" />
<solr:solr-server id="solrServer" url="http://localhost:8983/solr" />
<bean id="solrTemplate" class="org.springframework.data.solr.core.SolrTemplate">
<constructor-arg ref="solrServerFactory" />
</bean>
<solr:repositories base-package="com.test.solr.repositories" multicore-support="true" />
<bean id="solrServerFactory"
class="org.springframework.data.solr.server.support.MulticoreSolrServerFactory">
<constructor-arg ref="solrServer" />
<constructor-arg name="cores">
<list>
<value>program</value>
</list>
</constructor-arg>
</bean>

Spring Controller #ResponseBody text/xml response UTF-8 encoding issue

I have annotation based Spring Rest Service running on jetty web server(also tomcat).The controller code is :
#RequestMapping(method = RequestMethod.POST, value = { "/ssrfeed/exec/",
"/query/exec" }, consumes = { "application/xml", "text/xml",
"application/x-www-form-urlencoded" }, produces = {
"application/xml;charset=UTF-8", "text/xml;charset=UTF-8",
"application/x-www-form-urlencoded;charset=UTF-8" })
#ResponseBody
protected String getXmlFeed(HttpServletRequest request,
#PathVariable String serviceName, #RequestBody String xmlReq) {
//code....
return appXMLResponse;
}
The problem is that the response xml returned by Controller contains some characters like ä ö ü (Umlaute). The response when rendered on browser gives the parsing error :
XML Parsing Error: not well-formed
Location: //localhost:8083/MySerice/ssrfeed/exec/
Line Number 18111, Column 17:
<FIRST_NAME>Tzee rfista</FIRST_NAME>
----------------^
(a small triangle appear in place of ü)
The expected is : <FIRST_NAME>Tzeeürfista</FIRST_NAME>
I have tried a below solutions but issue is still there.
Tried using filters referring to solution given on technowobble
passed the charset to StringHttpMessageConverter property
<bean id="restTemplate" class="org.springframework.web.client.RestTemplate">
<property name="messageConverters">
<list>
<bean class="org.springframework.http.converter.json.MappingJacksonHttpMessageConverter">
<property name="supportedMediaTypes" value="application/json" />
</bean>
<bean class="org.springframework.http.converter.StringHttpMessageConverter">
<property name="supportedMediaTypes" value="text/xml;charset=UTF-8" />
</bean>
</list>
</property>
</bean>
ref link
Enabled the SetCharacterEncodingFilter in tomcat -web.xml
Changed the code to return ResponseEntity instead of String and removed #ResponseBody.
protected ResponseEntity<String> getXmlFeed(HttpServletRequest
request, #PathVariable String serviceName, #RequestBody String xmlReq) {
//line of code
HttpHeaders responseHeaders = new HttpHeaders();
responseHeaders.add("Content-Type", "application/xml; charset=utf-8");
return new ResponseEntity<String>(appXMLResponse, responseHeaders, HttpStatus.CREATED);
}
The 4th solution works But this being existing code I can't change method signature as it might impact existing clients of this service. Any ideas/pointers to solve this ?
in your dispatcher servlet context xml, you have to add a propertie. e.g.
<bean class = "org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<property name="messageConverters">
<array>
<bean class = "org.springframework.http.converter.StringHttpMessageConverter">
<property name="supportedMediaTypes" value = "text/plain;charset=UTF-8" />
</bean>
</array>
</property>
</bean>
Finally the issue is resolved. Here is what I did.
1. Used StringHttpMessageConverter's constructer for setting charset as :
<bean id="stringHttpMessageConverter"
class="org.springframework.http.converter.StringHttpMessageConverter">
<constructor-arg index="0" name="defaultCharset" value="UTF-8"/>
<property name="supportedMediaTypes">
<list>
<value>application/xml</value>
<value>text/xml</value>
<value>application/x-www-form-urlencoded</value>
</list>
</property>
</bean>
Also I removed the unnecessary spring3.0 and 3.1 jars from my project. These were not required but were lying there. (should have done earlier).
This solved the problem for me.
There is no such answer with what I've solved my encoding problem so I'll post it.
I've got Spring RestService running on Jetty. At response body part of data that was received from database had correct UTF-8 encoding, but data from .property file (with error and success messages) had incorrect encoding and was like äöü...
At first I checked encoding of .property file itself with File->Settings->Editor->Code Style-> File Encodings (in such way you could not only check but set encoding you need) - it was UTF-8.
Then I set response encoding #RequestMapping in my RestController:
#RequestMapping(value = "/category/{categoryId}", method = RequestMethod.DELETE, produces = { "application/json;**charset=UTF-8**" })
and set defaultCharset property for Jackson2:
<bean id="jsonConverter" class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter">
<property name="supportedMediaTypes" value="application/json;" />
<property name="prettyPrint" value="true" />
<property name="defaultCharset" value="UTF-8"/>
</bean>
No result.
But then I found that the problem could be solved by adding UTF-8 encoding to
PropertyPlaceholderConfigurer who grabs data from my .property file:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>classpath:app.properties</value>
<value>classpath:database.properties</value>
<value>classpath:ru.error.messages.properties</value>
<value>classpath:ru.success.messages.properties</value>
</list>
</property>
<property name="fileEncoding" value="UTF-8"/>
</bean>
... and the problem has gone )))

Spring 3.2 with MVC, ContentNegotation, REST and PDF Generator

Let's say, I have a REST styled controller mapping
#RequestMapping(value="users", produces = {MediaType.APPLICATION_JSON_VALUE})
public List<User> listUsers(#ReqestParams Integer offset, #ReqestParams Integer limit, #ReqestParams String query) {
return service.loadUsers(query, offset, limit);
}
Serving JSON (or even XML) is not an issue, this is easy using ContentNegotation and MessageConverters
<bean id="contentNegotiationManager" class="org.springframework.web.accept.ContentNegotiationManagerFactoryBean">
<property name="favorPathExtension" value="true" />
<property name="favorParameter" value="false" />
<property name="ignoreAcceptHeader" value="false" />
<property name="mediaTypes" >
<value>
html=text/html
json=application/json
xml=application/xml
</value>
</property>
</bean>
Now, I need to add support for PDF. Naturally, I want to use (Spring) MVC + REST as much as possible. Most examples I have found implement this with an explicit definition not using REST style, e.g.
#RequestMapping(value="users", produces = {"application/pdf"})
public ModelAndView listUsersAsPdf(#ReqestParams Integer offset, #ReqestParams Integer limit, #ReqestParams String query) {
List<User> users = listUsers(offset, limit, query); // delegated
return new ModelAndView("pdfView", users);
}
That works, but is not very comfortable because for every alternate output (PDF, Excel, ...) I would add a request mapping.
I have already added application/pdf to the content negotation resolver; unfortunately any request with a suffix .pdf or the Accept-Header application/pdf were be responded with 406.
What is the ideal setup for a REST/MVC style pattern to integrate alternate output like PDF?
You can create a WEB-INF/spring/pdf-beans.xml like below.
<bean id="listofusers" class="YourPDFBasedView"/>
And your controller method will return view name as listofusers.
#RequestMapping(value="users")
public ModelAndView listUsersAsPdf(#ReqestParams Integer offset, #ReqestParams Integer limit, #ReqestParams String query) {
List<User> users = listUsers(offset, limit, query); // delegated
return new ModelAndView("listofusers", users);
}
And you can use contentNegotiationViewResolver in this way:
<bean class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="order" value="1"/>
<property name="location" value="WEB-INF/spring/pdf-views.xml"/>
</bean>
<!--
View resolver that delegates to other view resolvers based on the content type
-->
<bean class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver">
<!-- All configuration is now done by the manager - since Spring V3.2 -->
<property name="contentNegotiationManager" ref="cnManager"/>
</bean>
<!--
Setup a simple strategy:
1. Only path extension is taken into account, Accept headers are ignored.
2. Return HTML by default when not sure.
-->
<bean id="cnManager" class="org.springframework.web.accept.ContentNegotiationManagerFactoryBean">
<property name="ignoreAcceptHeader" value="true"/>
<property name="defaultContentType" value="text/html" />
</bean>
For JSON: Create a generic JSON view resolver like below and register it as bean in context file.
public class JsonViewResolver implements ViewResolver {
/**
* Get the view to use.
*
* #return Always returns an instance of {#link MappingJacksonJsonView}.
*/
#Override
public View resolveViewName(String viewName, Locale locale) throws Exception {
MappingJacksonJsonView view = new MappingJacksonJsonView();
view.setPrettyPrint(true); // Lay the JSON out to be nicely readable
return view;
}
}
Same for XML:
public class MarshallingXmlViewResolver implements ViewResolver {
private Marshaller marshaller;
#Autowired
public MarshallingXmlViewResolver(Marshaller marshaller) {
this.marshaller = marshaller;
}
/**
* Get the view to use.
*
* #return Always returns an instance of {#link MappingJacksonJsonView}.
*/
#Override
public View resolveViewName(String viewName, Locale locale)
throws Exception {
MarshallingView view = new MarshallingView();
view.setMarshaller(marshaller);
return view;
}
}
and register above xml view resolver in context file like this:
<oxm:jaxb2-marshaller id="marshaller" >
<oxm:class-to-be-bound name="some.package.Account"/>
<oxm:class-to-be-bound name="some.package.Customer"/>
<oxm:class-to-be-bound name="some.package.Transaction"/>
</oxm:jaxb2-marshaller>
<!-- View resolver that returns an XML Marshalling view. -->
<bean class="some.package.MarshallingXmlViewResolver" >
<constructor-arg ref="marshaller"/>
</bean>
You can find more information at this link:
http://spring.io/blog/2013/06/03/content-negotiation-using-views/
Using all view resolver techniques, you can avoid writing duplicate methods in controller, such as one for xml/json, other for excel, other for pdf, another for doc, rss and all.
Knalli, if you replace #ResponseBody with ModelAndView(), you can achieve both the features.
Is there any reason you want to keep #ResponseBody ? I just want to know if I am missing anything, just want to learn.
Other option is to write HttpMessageConverters then:
Some samples are here.
Custom HttpMessageConverter with #ResponseBody to do Json things
http://www.javacodegeeks.com/2013/07/spring-mvc-requestbody-and-responsebody-demystified.html
This is working sample. I have configured contentnegotiationviewresolver for this, and give highest order. After that I have ResourceBundleViewResolver for JSTL and Tiles View, then XmlViewResolver for excelResolver, pdfResolver, rtfResolver. excelResolver, pdfResolver, rtfResolver. XmlViewResolver and ResourceBundleViewResolver works only with MAV only, but MappingJacksonJsonView and MarshallingView takes care for both MAV and #ResponseBody return value.
<bean id="contentNegotiatingResolver" class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver">
<property name="order"
value="#{T(org.springframework.core.Ordered).HIGHEST_PRECEDENCE}" />
<property name="mediaTypes">
<map>
<entry key="json" value="application/json" />
<entry key="xml" value="application/xml" />
<entry key="pdf" value="application/pdf" />
<entry key="xlsx" value="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" />
<entry key="doc" value="application/msword" />
</map>
</property>
<property name="defaultViews">
<list>
<!-- JSON View -->
<bean class="org.springframework.web.servlet.view.json.MappingJacksonJsonView" />
<!-- XML View -->
<bean class="org.springframework.web.servlet.view.xml.MarshallingView">
<constructor-arg>
<bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
<property name="classesToBeBound">
<list>
<value>Employee</value>
<value>EmployeeList</value>
</list>
</property>
</bean>
</constructor-arg>
</bean>
</list>
</property>
<property name="ignoreAcceptHeader" value="true" />
</bean>
<bean class="org.springframework.web.servlet.view.ResourceBundleViewResolver"
id="resourceBundleResolver">
<property name="order" value="#{contentNegotiatingResolver.order+1}" />
</bean>
<bean id="excelResolver" class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="location">
<value>/WEB-INF/tiles/spring-excel-views.xml</value>
</property>
<property name="order" value="#{resourceBundleResolver.order+1}" />
</bean>
<bean id="pdfResolver" class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="location">
<value>/WEB-INF/tiles/spring-pdf-views.xml</value>
</property>
<property name="order" value="#{excelResolver.order+1}" />
</bean>
<bean id="rtfResolver" class="org.springframework.web.servlet.view.XmlViewResolver">
<property name="location">
<value>/WEB-INF/tiles/spring-rtf-views.xml</value>
</property>
<property name="order" value="#{excelResolver.order+1}" />
</bean>
And our XMLViewResolver spring-pdf-views.xml looks like this.
<bean id="employees"
class="EmployeePDFView"/>
And EmployeePDFView will have code for generating pdf and writing pdf byte stream on Response object. This will resolve to rest url that will end with .pdf extension, and when you return MAV with "employees" id.

I want the current resource processed by MultiResourceItemReader available in beforeStep method

Is there any way to make the currentResource processed by MultiResourceItemReader to make it available in the beforeStep method.Kindly provide me a working code sample. I tried injected multiresourcereader reference in to stepexecutionlistener , but the spring cglib only accepts an interface type to be injected ,i dont know whether to use ItemReader or ItemStream interface.
The MultiResourceItemReader now has a method getCurrentResource() that returns the current Resource.
Retrieving of currentResource from MultiResourceItemReader is not possible at the moment. If you need this API enhancement, create one in Spring Batch JIRA.
Even if there is a getter for currentResource, it's value is not valid in beforeStep(). It is valid between open() and close().
it is possible, if you use a Partition Step and the Binding Input Data to Steps concept
simple code example, with concurrency limit 1 to imitate "serial" processing:
<bean name="businessStep:master" class="org.springframework.batch.core.partition.support.PartitionStep">
<property name="jobRepository" ref="jobRepository"/>
<property name="stepExecutionSplitter">
<bean class="org.springframework.batch.core.partition.support.SimpleStepExecutionSplitter">
<constructor-arg ref="jobRepository"/>
<constructor-arg ref="concreteBusinessStep"/>
<constructor-arg>
<bean class="org.spring...MultiResourcePartitioner" scope="step">
<property name="resources" value="#{jobParameters['input.file.pattern']}"/>
</bean>
</constructor-arg>
</bean>
</property>
<property name="partitionHandler">
<bean class="org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler">
<property name="taskExecutor">
<bean class="org.springframework.core.task.SimpleAsyncTaskExecutor">
<property name="concurrencyLimit" value="1" />
</bean>
</property>
<property name="step" ref="concreteBusinessStep"/>
</bean>
</property>
</bean>
<bean id="whateverClass" class="..." scope="step">
<property name="resource" value="#{stepExecutionContext['fileName']}" />
</bean>
example step configuration:
<job id="renameFilesPartitionJob">
<step id="businessStep"
parent="businessStep:master" />
</job>
<step id="concreteBusinessStep">
<tasklet>
<chunk reader="itemReader"
writer="itemWriter"
commit-interval="5" />
</tasklet>
</step>
potential drawbacks:
distinct steps for each file instead of one step
more complicated configuration
Using the getCurrentResource() method from MultiResourceItemReader, update the stepExecution. Example code as below
private StepExecution stepExecution;
#Override
public Resource getCurrentResource() {
this.stepExecution.getJobExecution().getExecutionContext().put("FILE_NAME", super.getCurrentResource().getFilename());
return super.getCurrentResource();
}
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
If you make the item you are reading implement ResourceAware, the current resource is set as it is read
public class MyItem implements ResourceAware {
private Resource resource;
//others
public void setResource(Resource resource) {
this.resource = resource;
}
public Resource getResource() {
return resource;
}
}
and in your reader, processor or writer
myItem.getResource()
will return the resource it was loaded from

Resources