I'm newbie in spring batch and I can't determinate what pattern for reader I would need to use. I need to create the class WSRequestClass and send it to SOAP web service.
public class WSRequestClass{
private String data1;
private String data2;
private String data3;
private String data4;
private List<ClassB> dataList;
}
To create WSRequestClass is necessary:
Read data1 and data2 from table A.
Read data3 and data4 from table B.
The List<ClassB> should be create from more complex flow. First I get data from query from table C, but the result of this query is a List<ClassA>. I need process each item of List<ClassA> and convert it to ClassB, where some attributes are calculated from ClassA. (Chunk pattern but without writer).
public class ClassA {
private Date date;
private BigDecimal amount1;
private BigDecimal amount2;
private String data;
//getters & setters
...
}
public class ClassB {
private Date date;
private BigDecimal amount1;
private BigDecimal amount2;
private BigDecimal amount3;
private BigDecimal amount4;
private String data1;
private String data2;
//getters & setters
...
}
I have found multiple examples for simples chunk pattern and tasklets, but none follows this structure. This job use java configuration and JdbcTemplate for queries. The development of the web service call it's done, my only issue is that I have to read from multiple tables and read efficiently the list, transform each item to ClassB and set to WsRequestClass.
Please guide me with the pattern to use, because common ItemReadernot work for me, and I don't know how implement the custom reader that allow me do what I want.
I think you're going about this wrong. There is a pattern in batch processing called the driving query pattern. In it, your reader reads essentially the keys for the objects. You then use processors to fill in the additional information. You can read more about this pattern in the Spring Batch documentation here: https://docs.spring.io/spring-batch/trunk/reference/html/patterns.html#drivingQueryBasedItemReaders
Related
We have model class saved in Redis as mentioned below:-
#Data
#NoArgsConstructor
#AllArgsConstructor
#RedisHash("book")
public class Book implements Serializable {
private static final long serialVersionUID = 2208852329346517265L;
#Id
private Integer bookID;
#Indexed
private String title;
#Indexed
private String authors;
private String averageRating;
private String isbn;
private String languageCode;
private String ratingsCount;
private BigDecimal price;
}
We have title and authors as our indexed property.
Now we wanted to search all the records from Redis by passing title and a list of authors using the spring crud repository as mentioned below.
public interface BookSpringRepository extends CrudRepository<Book, String> {
List<Book> findAllByTitleAndAuthors(String title, List<String> authors);
}
Service layer:-
#Override
public Optional<List<Book>> searchBooksByTitleAndAuthorNames(String title, List<String>
autherNames) {
return Optional.ofNullable(bookSpringRepository.findAllByTitleAndAuthors(title,
autherNames));
}
Here we are getting below exception
Unable to fetch data from Spring data Redis cache using List of Integer or
String.
Getting error while fetching - "Resolved
[org.springframework.core.convert.ConversionFailedException: Failed to convert from type
[java.lang.String] to type [byte] for value 'Ronak';
nested exception is java.lang.NumberFormatException: For input string: "Ronak"]."
We would not want to convert the list of string/integer to byte as it is a time-consuming process and as we tried took so much amount of time. Also when the results are retrieved we will again have to convert back to normal integer or string values.
The other option is to loop through the list and pass a single value at a time to the Redis crud repository and this time Redis crud repository is happy but that will be a loop call to Redis and network latency.
We cannot add ID attributes on authors' property as these can be duplicate records.
Does the spring crud repository support the LIKE query in search that way we can create a unique id having these authors' names and make put ID annotation on that new derived property to search the records using spring crud repository using LIKE or contains kind of query.
Any suggestions here are highly appreciated!!
Try to add serialization to your redis key and value. This might help :
https://medium.com/#betul5634/redis-serialization-with-spring-redis-data-lettuce-codec-1a1d2bc73d26
I have a database table which holds Metadata for documents. My task now is to get a list with documenttypes. The documenttypes are not unique in the database table but of course I want them to be in my list. The sql is very simple:
SELECT DISTINCT groupname, group_displayorder
FROM t_doc_metadata
ORDER BY group_displayorder;
I have learned that I can use projections to get a subset of fields from my entity DocMetadata. I solved this as follows. My Entity:
#Entity
#Table(name="T_DOC_METADATA")
#Data
public class DocMetadata {
..............
#Column(nullable=false)
private String displayname;
#Column(nullable=false)
private Integer displayorder;
#Column(nullable=false)
private String groupname;
#Column(name="GROUP_DISPLAYORDER",
nullable=false)
private Integer groupDisplayorder;
#Column(name="METADATA_CHANGED_TS",
nullable=false,
columnDefinition="char")
private String metadataChangedTimestamp;
..........
}
My inteface for projection:
public interface GroupnameAndOrder {
String getGroupname();
Integer getGroupDisplayorder();
void setGroupname(String name);
void setGroupDisplayorder(int order);
}
Now I thought I'd be extraordinary clever by adding these lines to my repository:
#Query("select distinct d.groupname, d.groupDisplayorder from DocMetadata d order by d.groupDisplayorder")
public List<GroupnameAndOrder> findSortedGroupnames();
Sadly, when iterating over the result list and calling getGroupname() the result is null.
So I changed the lines in my repository according to the documentation:
public List<GroupnameAndOrder> findBy();
Now I get the groupnames but of course they are not unique now. So it doesn't solve my problem.
Is there any way to receive a ordered list with unique groupnames?
You are trying to be too clever. Instead just write the proper find method and return the GroupnameAndOrder. Spring Data JPA will then only retrieve what is needed for the projection.
Something like this should do the trick.
List<GroupnameAndOrder> findDistinctByOrderByGroupDisplayorder();
Using spring-data-jpa and working on getting data out of table where there are about a dozen columns which are used in queries to find particular rows, and then a payload column of clob type which contains the actual data that is marshalled into java objects to be returned.
Entity object very roughly would be something like
#Entity
#Table(name = "Person")
public class Person {
#Column(name="PERSON_ID", length=45) #Id private String personId;
#Column(name="NAME", length=45) private String name;
#Column(name="ADDRESS", length=45) private String address;
#Column(name="PAYLOAD") #Lob private String payload;
//Bunch of other stuff
}
(Whether this approach is sensible or not is a topic for a different discussion)
The clob column causes performance to suffer on large queries ...
In an attempt to improve things a bit, I've created a separate entity object ... sans payload ...
#Entity
#Table(name = "Person")
public class NotQuiteAWholePerson {
#Column(name="PERSON_ID", length=45) #Id private String personId;
#Column(name="NAME", length=45) private String name;
#Column(name="ADDRESS", length=45) private String address;
//Bunch of other stuff
}
This gets me a page of NotQuiteAPerson ... I then query for the page of full person objects via the personIds.
The hope is that in not using the payload in the original query, which could filtering data over a good bit of the backing table, I only concern myself with the payload when I'm retrieving the current page of objects to be viewed ... a much smaller chunk.
So I'm at the point where I want to map the contents of the original returned Page of NotQuiteAWholePerson to my List of Person, while keeping all the Paging info intact, the map method however only takes a Converter which will iterate over the NotQuiteAWholePerson objects ... which doesn't quite fit what I'm trying to do.
Is there a sensible way to achieve this ?
Additional clarification for #itsallas as to why existing map() will not suffice..
PageImpl::map has
#Override
public <S> Page<S> map(Converter<? super T, ? extends S> converter) {
return new PageImpl<S>(getConvertedContent(converter), pageable, total);
}
Chunk::getConvertedContent has
protected <S> List<S> getConvertedContent(Converter<? super T, ? extends S> converter) {
Assert.notNull(converter, "Converter must not be null!");
List<S> result = new ArrayList<S>(content.size());
for (T element : this) {
result.add(converter.convert(element));
}
return result;
}
So the original List of contents is iterated through ... and a supplied convert method applied, to build a new list of contents to be inserted into the existing Pageable.
However I cannot convert a NotQuiteAWholePerson to a Person individually, as I cannot simply construct the payload... well I could, if I called out to the DB for each Person by Id in the convert... but calling out individually is not ideal from a performance perspective ...
After getting my Page of NotQuiteAWholePerson I am querying for the entire List of Person ... by Id ... in one call ... and now I am looking for a way to substitute the entire content list ... not interively, as the existing map() does, but in a simple replacement.
This particular use case would also assist where the payload, which is json, is more appropriately persisted in a NoSql datastore like Mongo ... as opposed to the sql datastore clob ...
Hope that clarifies it a bit better.
You can avoid the problem entirely with Spring Data JPA features.
The most sensible way would be to use Spring Data JPA projections, which have good extensive documentation.
For example, you would first need to ensure lazy fetching for your attribute, which you can achieve with an annotation on the attribute itself.
i.e. :
#Basic(fetch = FetchType.LAZY) #Column(name="PAYLOAD") #Lob private String payload;
or through Fetch/Load Graphs, which are neatly supported at repository-level.
You need to define this one way or another, because, as taken verbatim from the docs :
The query execution engine creates proxy instances of that interface at runtime for each element returned and forwards calls to the exposed methods to the target object.
You can then define a projection like so :
interface NotQuiteAWholePerson {
String getPersonId();
String getName();
String getAddress();
//Bunch of other stuff
}
And add a query method to your repository :
interface PersonRepository extends Repository<Person, String> {
Page<NotQuiteAWholePerson> findAll(Pageable pageable);
// or its dynamic equivalent
<T> Page<T> findAll(Pageable pageable, Class<T>);
}
Given the same pageable, a page of projections would refer back to the same entities in the same session.
If you cannot use projections for whatever reason (namely if you're using JPA < 2.1 or a version of Spring Data JPA before projections), you could define an explicit JPQL query with the columns and relationships you want, or keep the 2-entity setup. You could then map Persons and NotQuiteAWholePersons to a PersonDTO class, either manually or (preferably) using your object mapping framework of choice.
NB. : There are a variety of ways to use and setup lazy/eager relations. This covers more in detail.
Say I have the following Collections
public #Data class Customer {
#Id
private String id;
private String firstName;
private String lastName;
#DBRef
private List<Address> addressList= new ArrayList<Address>();
}
and
public #Data class Address {
#Id
private String id;
private String address;
private String type;
private String customerID;
}
And each Customer has multiple addresses, and I have implemented MongoRepository. Saving customer for the First time is working pretty well customerRepo.save(customerObject) and before calling the save I am persisting multiple Address Objects and then setting those to the addressList.
Next time when I am updating the same document and want to add a New set of Address to the existing list it is overwriting the whole addressList array. So basically what I have to do now to set new address like thisexistingCustomerObject.getAddressList().addAll(my new List of address) if there are thousand(or more than thousand) of elements or I am slicing the addressList array the following procedure won't be a good idea. My question is what is the best way to achieve this scenario? say if I don't want to use MongoTemplate. Is it possible Just using the MongoRepository
I don't think you can do it in that way. Previously i had the same situation, and I tried the following
1.org.springframework.core.convert.converter.Converter even I have managed to manipulate the DBObject but functions like $push or $set(wrapping them under key) does not work over there.
2.AbstractMongoEventListener by overriding onBeforeSave but Object manipulation was not taking place during save.
However you can try altering the mentioned
you can try override MongoRepository save method, It would better if someone point to the right direction.
Otherwise for my scenario I had to create Custom repository(To update and delete document) which is working parallel with MongoRepository (for Insert and retrieve data/document), but I believe thats an ugly workaround. There has to be a cleaner way to do it.
I have some problem with Morphia.
Could someone help me?
I am writing web-project on Spring + MongoDB about movies and celebrities.
I have entity class Genre:
#Entity(value="genres")
public class Genre implements IGenre {
#Id
#Indexed
private ObjectId id;
#Indexed
private String name;
private String description;
private long quantity;
private Set <IMovie> movies;
//getters and setters
}
And entity class Movie:
#Entity(value="movies")
public class Movie implements IMovie {
#Id
#Indexed
private ObjectId id;
#Indexed
private String originalTitle;
private String year;
private Set <IGenre> genres;
// getters and setters
}
I have 30 genres. And for example one of them: Comedy.
Also I have 250 000 comedies.
And now I want to do movie pagination by genre = comedy.
How I can get only 20 records from all comedies.
If I use #Embedded or #Reference annotation I will still get the entire list at once. And it's to big for use it in controllers.
You should change your data schema for doing such a query. The schema you use has a circular dependency, in your Genre entity you are holding Movies entity and in movies you hold Genre. Also holding all of the movies according to genre is not easy to query. If I were you I would use such a schema.
#Entity(noClassnameStored = true) // you wouldn't have any problem when you change the class name or package of your class if you don't store the classname through this annotation
public class Movie implements IMovie {
#Id
#Indexed
private ObjectId id;
#Indexed
private String originalTitle;
private String year;
private Set <String> genres; // unique identifier of the genres instead of embedding the whole genre entity
// getters and setters
}
So having such a schema, you can retrieve the movies having a particular genre by writing a simple $in query through genres field. Example query for your case:
datastore.find(Movie.class).field("genres").in(Lists.newArrayList("comedy")).limit(20).asList;
In the below web-page of mongo you can find suggestions about how to design your schema according to diffrerent scenarios.
http://docs.mongodb.org/manual/core/data-modeling/#data-modeling-patterns-and-examples
I'm not too familiar with Mongo, but it looks like you would need to implement a custom query here. Whatever you do, you need to pass a start for your page, as well as a page size (20 in your case).
You can do pagination in Morphia by combining .offset(page_start) and .limit(page_size) on a query. So first you would create a query to get movies that belong to a certain genre, and then apply the pagination.
Looks like it is handled using repositories in Spring http://static.springsource.org/spring-data/data-mongodb/docs/1.0.0.RELEASE/reference/html/#repositories.special-parameters
You'd use a Pageable implementation to pass paging data, without worrying about doing the offset and limit calls yourself. There's an example of "Web pagination" further in the doc.
Hope this helps!