I am writing a Spring Batch application with the following workflow:
Read some items of type A (using a FlatFileItemReader<A>).
Process an item, transforming it from A to B.
Write the processed items of type B (using a JdbcBatchItemWriter<B>)
Eventually, I should call an external service (a RESTful API, but it could be a SimpleMailMessageItemWriter<A>) using data from the source type A.
How can I configure such a workflow?
So far, I have found the following workaround:
Configuring a CompositeItemWriter<B> which delegates to:
The actual ItemWriter<B>
A custom ItemWriter<B> implementation which converts B back to A and then writes an A
But this is a cumbersome solution because it forces me to either:
Duplicate processing logic: from A to B and back again.
Sneakily hide some attributes from the source object A inside B, polluting the domain model.
Note: since my custom item writer for A needs to invoke an external service, I would like to perform this operation after B has been successfully written.
Here are the relevant parts of the batch configuration code.
#Bean
public Step step(StepBuilderFactory steps, ItemReader<A> reader, ItemProcessor<A, B> processor, CompositeItemWriter<B> writer) {
return steps.get("step")
.<A, B>chunk(10)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
#Bean
public CompositeItemWriter<B> writer(JdbcBatchItemWriter<B> jdbcBatchItemWriter, CustomItemWriter<B, A> customItemWriter) {
return new CompositeItemWriterBuilder<B>()
.delegates(jdbcBatchItemWriter, customItemWriter)
.build();
}
For your use case, I would encapsulate A and B in a wrapper type, such AB:
class AB {
private A originalItem;
private B transformedItem;
}
With that, you would have: ItemReader<A>, ItemProcessor<A, AB> and ItemWriter<AB>. The processor creates instances of AB in which it keeps a reference to the original item. The writer can then get access to both types and delegate to the JdbcBatchItemReader<B> and SimpleMailMessageItemWriter<A> as needed, something like:
class ABItemWriter implements ItemWriter<AB> {
private JdbcBatchItemWriter<B> jdbcBatchItemWriter;
private SimpleMailMessageItemWriter mailMessageItemWriter;
// constructor with delegates
#Override
public void write(List<? extends AB> items) throws Exception {
jdbcBatchItemWriter.write(getBs(items));
mailMessageItemWriter.write(getAs(items)); // this would not be called if the jdbc writer fails
}
}
The methods getAs and getBs would extract items of type A/B from AB. Encapsulation for the win! BTW, a Java record is a good option for type AB.
Related
This seems to be a missing part in the documentation of Vaadin...
I call an API to get data in my UI like this:
#Override
public URI getUri(String url, PageRequest page) {
return UriComponentsBuilder.fromUriString(url)
.queryParam("page", page.getPageNumber())
.queryParam("size", page.getPageSize())
.queryParam("sort", (page.getSort().isSorted() ? page.getSort() : ""))
.build()
.toUri();
}
#Override
public Mono<Page<SomeDto>> getDataByPage(PageRequest pageRequest) {
return webClient.get()
.uri(getUri(URL_API + "/page", pageRequest))
.retrieve()
.bodyToMono(new ParameterizedTypeReference<>() {
});
}
In the Vaadin documentation (https://vaadin.com/docs/v10/flow/binding-data/tutorial-flow-data-provider), I found an example with DataProvider.fromCallbacks but this expects streams and that doesn't feel like the correct approach as I need to block on the requests to get the streams...
DataProvider<SomeDto, Void> lazyProvider = DataProvider.fromCallbacks(
q -> service.getData(PageRequest.of(q.getOffset(), q.getLimit())).block().stream(),
q -> service.getDataCount().block().intValue()
);
When trying this implementation, I get the following error:
org.springframework.core.codec.CodecException: Type definition error: [simple type, class org.springframework.data.domain.Page]; nested exception is com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.springframework.data.domain.Page` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
grid.setItems(lazyProvider);
I don't have experience with vaadin, so i'll talk about the deserialization problem.
Jackson needs a Creator when deserializing. That's either:
the default no-arg constructor
another constructor annotated with #JsonCreator
static factory method annotated with #JsonCreator
If we take a look at spring's implementations of Page - PageImpl and GeoPage, they have neither of those. So you have two options:
Write your custom deserializer and register it with the ObjectMapper instance
The deserializer:
public class PageDeserializer<T> extends StdDeserializer<Page<T>> {
public PageDeserializer() {
super(Page.class);
}
#Override
public Page<T> deserialize(JsonParser p, DeserializationContext ctxt) throws IOException, JacksonException {
//TODO implement for your case
return null;
}
}
And registration:
SimpleModule module = new SimpleModule();
module.addDeserializer(Page.class, new PageDeserializer<>());
objectMapper.registerModule(module);
Make your own classes extending PageImpl, PageRequest, etc. and annotate their constructors with #JsonCreator and arguments with #JsonProperty.
Your page:
public class MyPage<T> extends PageImpl<T> {
#JsonCreator
public MyPage(#JsonProperty("content_prop_from_json") List<T> content, #JsonProperty("pageable_obj_from_json") MyPageable pageable, #JsonProperty("total_from_json") long total) {
super(content, pageable, total);
}
}
Your pageable:
public class MyPageable extends PageRequest {
#JsonCreator
public MyPageable(#JsonProperty("page_from_json") int page, #JsonProperty("size_from_json") int size, #JsonProperty("sort_object_from_json") Sort sort) {
super(page, size, sort);
}
}
Depending on your needs for Sort object, you might need to create MySort as well, or you can remove it from constructor and supply unsorted sort, for example, to the super constructor. If you are deserializing from input manually you need to provide type parameters like this:
JavaType javaType = TypeFactory.defaultInstance().constructParametricType(MyPage.class, MyModel.class);
Page<MyModel> deserialized = objectMapper.readValue(pageString, javaType);
If the input is from request body, for example, just declaring the generic type in the variable is enough for object mapper to pick it up.
#PostMapping("/deserialize")
public ResponseEntity<String> deserialize(#RequestBody MyPage<MyModel> page) {
return ResponseEntity.ok("OK");
}
Personally i would go for the second option, even though you have to create more classes, it spares the tediousness of extracting properties and creating instances manually when writing deserializers.
There are two parts to this question.
The first one is about asynchronously loading data for a DataProvider in Vaadin. This isn't supported since Vaadin has prioritized the typical case with fetching data straight through JDBC. This means that you end up blocking a thread while the data is loading. Vaadin 23 will add support for doing that blocking on a separate thread instead of keeping the UI thread blocked, but it will still be blocking.
The other half of your problem doesn't seem to be directly related to Vaadin. The exception message says that the Jackson instance used by the REST client isn't configured to support creating instances of org.springframework.data.domain.Page. I don't have direct experience with this part of the problem, so I cannot give any advice on exactly how to fix it.
Suppose I have a caller to distribute work to multiple async tasks:
public class Caller{
public boolean run() {
for (int i = 0: i< 100; i++) {
worker.asyncFindOrCreate(entites[i]);
}
return true;
}
public class Worker{
#Autowired
Dao dao;
#Async
public E asyncFindOrCreate(User entity) {
return dao.findByName(entity.getName).elseGet(() -> dao.save(entity));
}
}
If we have 2 same entities:
with the synchronized method, the first one will be created and then the second one will be retrieved from the existing entity;
with async, the second entities might pass the findByName and go to save because the first entity hasn't been saved yet, which cause the save of the second entity throws unique identifier error.
Is there a way to add some fault tolerance mechanic to have some features like retry and skipAfterRetry, in particular for database operations.
In this special case you should convert your array to a map. Use the name property as a key, so there will be no duplicated entries.
However, if this method also can be called by multiple threads (ie. it's in a web-server) or there are multiple instances running it's still not fail-safe.
In generic, you should let the DB to check the uniqueness. There is no safest/easiest way to do that. Put the save method inside a try-catch block and check/handle the unique identifier exception.
I am reading up on AsyncCassandraOperations to perform async inserts to improve performance based on another post here. But I am unable to find a lot of help on google or spring data documentation.
Previously I was using Cassandra Repository for all data extraction and insert/updates which I found to be super slow. As per recommendation I am now using AsyncCassandraOperations for the insert operation alone, but it wont let me. I encounter required a bean of type 'org.springframework.data.cassandra.core.AsyncCassandraOperations' error.
What would be the correct way to use AsyncCassandraOperations please?
#Autowired private MyRepository repository_name;
#Autowired private AsyncCassandraOperations acops;
public void persist(List<POJO> l_POJO)
{
System.out.println("Enter Persist: "+new java.util.Date());
List<l_POJO> l_POJO_stale = repository_name.findBycol1AndStale("sample",false);
l_POJO_stale.forEach(s -> s.setStale(true));
l_POJO_stale.forEach(s -> acops.update(s));
try
{
acops.insert(l_POJO);
}
catch (Exception e)
{
System.out.println("Error in persisting new data");
}
}
Don't know whether spring boot is used, if so the AsyncCassandraOperations(AsyncCassandraTemplate is the implementation class) should be created automatically.
If the error shows you need an AsyncCassandraOperations bean, the straight way is to create one as shown below.
#Bean
AsyncCassandraTemplate asyncCassandraTemplate(Session session) {
return new AsyncCassandraTemplate(session);
}
Since you are using Spring data Repository interface, you can alse use the ReactiveCrudRepository interface to update or insert entity objects to Cassandra, which is shown in this spring data example project , as an alternative way to using the AsyncCassandraTemplate class.
In the case of using ReactiveCrudRepository and regarding what you want to do, your code needs the following changes.
change the return type of WRRepository.findByCol1AndCol2AndCol3(String, boolean, String) from List<WRpojo> to Flux<WRpojo> , in order to fully utilize the reactive functionality.
change the return type of persist(List<WRpojo>) from boolean to Mono<Void> , making the result non-blocking too.
change your persist(List<WRpojo>) to the following.
public Mono<Void> persist(List<WRpojo> l_wr) {
Flux<WRpojo> l_old_wr = objWRRepository.findByCol1AndCol2AndCol3("1", false, "2").doOnNext(s -> s.setStale(true));
return objWRRepository.saveAll(l_old_wr).thenMany(objWRRepository.saveAll(l_wr)).then();
}
In reactive programming, basically we don't block any code, this means that somewhere the returned Mono<Void> should be subscribed somewhere downstream, if you do want to block and wait for all operations complete, you can call block() on Mono<Void> , which is not recommended.
I have a gradle multiproject with 2 subprojects trying to emulate an hexagonal architecture :
rest-adapter
application layer
I don't want the application services to expose the domain models and do'nt want to force a specific representation as output. So I would like something like application services consume 2 args (a command and something) and return a T. The client configures the service.
The rest adapter doesn't ave access to the domain model, so I can't return the domain models and let the adapter creates its representation.
What about the something. I tried :
have a signature <T> List<T> myUseCase(Command c, Function<MyDomainModel, T> fn). The application layer is the owner of transformations functions (because the signature uses MyDomainModel) and exposes a dictionnary of function. So the rest controller references one of the Fn. It works. And I'm searching of a better way. More elegant way if it exists.
have a signature <T> List<T> myUseCase(Command c, FnEnum fn) For each enum I have associated a Function. With this, I found the signature more elegant : the consumer provides which transformation it wants from an enum. But doesn't work cause the generic method doesn't compile. The cannot be resolved. Currently, I didn't find a way.
something with java 8 consumer or supplier or something else but I failed to wrap my head around.
I'm feeling there's a more elegant solution for this kind of problem : a service which accepts a function that transforms and build an output that the client provides.
I think that what you need to implement is the so called "Data Transformer" pattern.
Imagine that you have a use case that returns a certain domain object (for example "User"), but you shouldn't expose domain to clients. And you want every client to choose the format of the returned data.
So you define a data transformer interface for the domain object:
public interface UserDataTransformer {
public void write ( User user );
public String read();
}
For every output format your clients need you define a class implementing the interface. For example if you want to represent the User in XML format:
public class UserXMLDataTransformer implements UserDataTransformer {
private String xmlUser;
#Override
public void write(User user) {
this.xmlUser = xmlEncode ( user );
}
private String xmlEncode(User user) {
String xml = << transform user to xml format >>;
return xml;
}
#Override
public String read() {
return this.xmlUser;
}
}
Then you make your application service depends on the data trasnsformer interface, you inject it in the constructor:
public class UserApplicationService {
private UserDataTransformer userDataTransformer;
public UserApplicationService ( UserDataTransformer userDataTransformer ) {
this.userDataTransformer = userDataTransformer;
}
public void myUseCase ( Command c ) {
User user = << call the business logic of the domain and construct the user object you wanna return >> ;
this.userDataTransformer.write(user);
}
}
And finally, the client could look something like this:
public class XMLClient {
public static void main ( String[] args ) {
UserDataTransformer userDataTransformer = new UserXMLDataTransformer();
UserApplicationService userService = new UserApplicationService(userDataTransformer);
Command c = << data input needed by the use case >>;
userService.myUseCase(c);
String xmlUser = userDataTransformer.read();
System.out.println(xmlUser);
}
}
I've consider that the output is a String, but you could use generics maybe to return any type you want.
I haven't mentioned it, but this approach injecting the transformer into the application service follows the "port and adapters" pattern. The transformer interface would be the port, and every class implementing it would be an adapter for the desired format.
Also, this was just an example. You can use a dependency injection framework like Spring in order to create the component instances and wire them all. And also you should use the composition root pattern to do it.
Hope this example helped.
I'm feeling there's a more elegant solution for this kind of problem : a service which accepts a function that transforms and build an output that the client provides.
You are sending data across the boundary between the application and the REST layer (and presumably between the application and the REST consumer); it may be useful to think about messaging patterns.
For example, the application can define a service provider interface that defines a contract/protocol for accepting data from the application.
interface ResponseBuilder {...}
void myUseCase(Command c, ResponseBuilder builder)
The REST adapter provides an implementation of the ResponseBuilder that can take the inputs and generate some useful data structure from them.
The response builder semantics (the names of the functions in the interface) might be drawn from the domain model, but the arguments will normally be either primitives or other message types.
CQS would imply that a query should return a value; so in that case you might prefer something like
interface ResponseBuilder<T> {
...
T build();
}
<T> T myUseCase(Command c, ResponseBuilder<T> builder)
If you look carefully, you'll see that there's no magic here; we've simply switched from having a direct coupling between the application and the adapter to having an indirect coupling with the contract.
EDIT
My first solution is using a Function<MyDomainModel, T> which is a bit different from your ResponseBuilder ; but in the same vein.
It's almost dual to it. You'd probably be a little bit better off with a less restrictive signature on myUseCase
<T>
List<T> myUseCase(Command c, Function<? super MyDomainModel, T> fn)
The dependency structure is essentially the same -- the only real difference is what the REST adapter is coupled to. If you think the domain model is stable, and the output representations are going to change a lot, then the function approach gives you the stable API.
I suspect that you will find, however, that the output representations stabilize long before the domain model does, in which case the ResponseBuilder approach will be the more stable choice.
I use spring batch for processing a file. The configuration of all components is made programatically.
I have a job that contains several TaskletSteps:
#Bean
#Named(SEEC_JOB)
public Job seecJob() {
return jobBuilderFactory.get(SEEC_JOB).start(seecMoveToWorkingStep()).next(seecLoadFileStep())
.on(ExitStatus.COMPLETED.getExitCode()).to(seecFlowMoveToArchiveOk()).from(seecLoadFileStep())
.on(ExitStatus.FAILED.getExitCode()).to(seecFlowMoveToArchiveKo()).end().build();
}
My question focus on seecLoadFileStep(), the detail bellow:
#Bean
public TaskletStep seecLoadFileStep() {
TaskletStep build = stepBuilderFactory.get(SEEC_LOAD_FILE_STEP)
.<SeecMove, SeecMove>chunk(cormoranProperties.seec.batchSize.get()).reader(seecItemReader())
.writer(seecItemWriter()).build();
return build;
}
I would like to throw a specific exception if a reading error hapens (by reading error I mean: the file is corrupted for example or it is wrong, absent xml tag...).
I have been reading spring batch doc and I think ItemReadListener is my guy:
public interface ItemReadListener<T> extends StepListener {
void beforeRead();
void afterRead(T item);
void onReadError(Exception ex);
}
but, I don't know how to use it! I have tried doing my seecItemReader() implements this interface but onReadError method is never called.
I don't know how to declare/register in the taskletStep the ItemReadListener.
Here a bit of spring doc:
Any class that implements one of the extensions of StepListener (but
not that interface itself since it is empty) can be applied to a step
via the listeners element. The listeners element is valid inside a
step, tasklet or chunk declaration. It is recommended that you declare
the listeners at the level which its function applies, or if it is
multi-featured (e.g. StepExecutionListener and ItemReadListener) then
declare it at the most granular level that it applies (chunk in the
example given).
An ItemReader, ItemWriter or ItemProcessor that itself implements one
of the StepListener interfaces will be registered automatically with
the Step if using the namespace element, or one of the the
*StepFactoryBean factories. This only applies to components directly injected into the Step: if the listener is nested inside another
component, it needs to be explicitly registered (as described above).
Could you please help me?
Thanks in advance!
As I guessed it was easier than I thougth, for registering programatically the ItemReadListener is via listener method in the tasklet configuration:
#Bean
public TaskletStep seecLoadFileStep() {
TaskletStep build = stepBuilderFactory.get(SEEC_LOAD_FILE_STEP)
.<SeecMove, SeecMove>chunk(cormoranProperties.seec.batchSize.get()).reader(seecItemReader()).listener(seecItemReaderListener())
.writer(seecItemWriter()).build();
return build;
}
And now the onError method is called when an Exception happens.