Today we have a web app and a legacy API exposing some of REST endpoints to get booking of client. The booking are fetched with a SOAP operation, and the returned response for a specific type of booking differs from the other types. Moreover, these calls are latent and cause a long wait, especially when a client has more than 5 bookings. The next diagram shows a simple interaction between components:
legacy app
The idea of this project is to copy the existent data (clients and booking) in a new schema and get a copy of soap response for each client and booking. The next diagram shows a simple interaction between components with spring batch:
new app
My goal is to use Spring batch to:
copy client an booking tables
use the couple booking type/references for each client to call SOPA operations
persist the result in new booking information table
clients = clientRepository.findBy(id)
foreach client in clients
bookings = bookingRepository.findBy(id)
foreach booking in bookings
call soap with (booking.reference, booking.type)
map newBooking
persist newBooking with client.id, booking.reference
I am new to Spring and I don't know how to do these operations with parameterized query in a repository class and nested with SOAP calls.
One option to implement the pseudo code of your solution in Spring Batch is to create a chunk-oriented step defined as follows:
The item reader reads client IDs
An item processor fetches bookings for each client
The item writer writes data to both tables
This pattern has a name, the "driving query pattern", which is documented here: Common Batch Patterns -> Driving Query Based ItemReaders.
Related
The spring boot application that I am working on
pools 1000 messages from table X [ This table X is populated by another service s1]
From each message get the account number and query table Y to get additional information about account.
I am using spring integrating to pool messages from table X and reading additional information for account, I am planning to use Spring JDBC.
We are expecting about 10k messages very day.
Is above approach, to query table Y for each message, a good approach ?
No, that indeed not. If all of that data is in the same database, consider to write a proper SELECT to join those tables in a single query performed by that source polling channel adapter.
Another approach is to implement a stored procedure which will do that job for you and will return the whole needed data: https://docs.spring.io/spring-integration/reference/html/jdbc.html#stored-procedures.
Although if the memory for that number of records to handle at once is a limit in your environment or you don't care how fast all of them are processed, then indeed an integration flow with parallel processing of splitted polling result is OK. For that goal you can use a JdbcOutboundGateway as a service in your flow instead of playing with plain JdbcTemplate: https://docs.spring.io/spring-integration/reference/html/jdbc.html#jdbc-outbound-gateway
I've already made an action of type query that calls an http endpoint and return a list of results.
Then i should insert this resut into the postgresql (i suppose through a mutation).
So, how can i join this insert mutation to the previus query result, and eventually apply some custom logic (eg: not insert the already present records)
I was looking into this myself a couple of days ago, and my takeaway so far was that this is currently not possible. You would still have to write a small service (e.g. aws lambda) that calls your action and plugs the result into the mutation. That is also where you can apply your business logic.
It would be a great feature to have, in order to connect two APIS directly together or even just do data transfers from one place to another.
The new Rest transformers released in 2.1 at least make it easier and faster to integrate with existing APIs, so all you need to do is the plumbing now
I have the following scenario. In a database A I have a table with huge amount of records (several millions); these records increase day by day very rapidly (also 100.000 records at day).
I need to fetch these records, check if these records are valid and import them in my own database. At the first interaction I should take all the stored records. Then I can take only the new records saved. I have a timestamp column I can use for this filter but I can't figure how to create a JpaPagingItemReader or a JdbcPagingItemReader and pass the dynamic filter based on the date (e.g. select all records where timestamp is greater than job last execution date)
I'm using spring boot, spring data jpa and spring batch.I'm configuring the Job instance in chunks with dimension 1000. I can also use a paging query (is it useful if I use chunks?)
I have a micro service (let's call this MSA) with all the business logic needed to check if records are valid and insert the valid records.
I have another service on a separate server. This service contains all the batch operation (let's call this MSB).
I'm wondering what is the best approach to the batch. I was thinking to these solutions:
in MSB I duplicate all the entities, repositories and services I use in the MSA. Then in MSB I can make all needed queries
in MSA I create all the rest API needed. The ItemProcessor of MSB will call these rest API to perform checks on items to be processed and finally in the ItemWriter I'll call the rest API for saving data
The first solution would avoid the http calls but it forces me to duplicate all repositories and services between the 2 micro services. Sadly I can't use a common project where to place all the common objects.
The second solution, on the other hand, would avoid the code duplication but it would imply a lot of http calls (above all in the ItemProcessor to check if an item is valid or less).
Do you have any other suggestion? Is there a better approach?
Thank you
Angelo
I have two Frontends consuming JSON from two different Backends using the JSON Web Token. These backends act on the same database.
In the db for example I have the Driver, Customer and Trip tables. The customer or the driver can cancel a trip only if it has not been canceled beforehand by one of them. Some transactions are recorded during a cancellation.
How to prevent having a double execution in this case when simultaneously, the customer and the driver launch a request for trip cancellation?
Am usin' Spring Boot (RESTful) and Spring JPA.
Any help will be greatly appreciated.
Edit:
Assuming these backends are A & B, Customer is requesting cancellation from the backend A, and Driver from B.
Use optimistic locking. Your code would look as follows:
#Entity
public class Trip {
#Version
#NotNull
private Long version;
...
}
It works as follows. Each change modifies the version. Suppose two users (or two services) loaded the same version of the trip. Now they both try to cancel it, i.e. they both try to modify it. Besides changes they both send the version. The JPA checks if the version in the update statement is the same as in the database. So the first request wins and will be executed. During the execution the version will be incremented.
Now the 2nd request arrives and wants also to cancel the trip. The JPA will see that the version attribute in the update statement is older (less) than the version value in database. Thus the 2nd request will not be executed and an OptimisticLockException will be thrown.
You can catch this exception and inform the user that the data were change in the meanwhile and suggest user to reload the data. The user reloads the data and sees that the trip has already been cancelled.
I have created simple CRUD api using Spring Data JPA in my Spring boot application.My Post method in the controller looks like below:-
#RequestMapping(value = "/article", method = RequestMethod.POST, produces = "application/json")
public Article createArticle(#RequestBody Article article) {
return service.createArticle(article);
}
Service Method is as follows:-
#Override
public Article createArticle(Article articleModel) {
return repository.save(articleModel);
}
My JsonPayload looks like below:
{
"article_nm":"A1",
"article_identifier":"unique identifier"
}
Now I want to make my POST request as Idempotent so that even if i got the json payload with the same article_identifier again It would not create a record in DB.
I can't do any scheme/constraint change in database and article_identifier field is also not primary key in table.
I understand that first I can check in database and return the saved record in response if it already exists but here if multiple request (original and duplicate request) comes at same time, both will check in database and would not find any record with that identifier and will create 2 record (one for each). Also as it's a distributed application how can i maintain the consistency across multiple database transactions.
How can I use some locking mechanism so that there would not be 2 records with same article_identifier ever. Can somebody please suggest some refers how to implement it in Spring boot ?
Idempotency in this case is needed to solve the post-back (or double post request). The simples way would be just to check at the service level whether a post with a given information exists (as you pointed out). You can use repository.exists() variations for that.
I understand that first I can check in database and return the saved record in response if it already exists
As for
if multiple request (original and duplicate request) comes at same time, both will check in database and would not find any record with that identifier and will create 2 record (one for each)
You need to isolate the transactions from each other if it is a single database (I know you said it is not, but I'm trying to explain my reasoning so bear with me). For that spring has the following anotation: #Transactional(isolation = Isolation.SERIALIZABLE). Although in this case #Transactional(isolation = Isolation.REPEATABLE_READ) would be enough.
Also as it's a distributed application how can i maintain the consistency across multiple database transactions.
How is it distributed? You first need to think about the database. Is it a master-slave mysql / postgress / mongodb ? Is it some weird globally distributed system? Assuming it is the traditional master-slave setup then the write transaction will be handled by the master (to my knowledge all the selects belonging to the transaction will also be there) so there should be no problem. However the answer can only really be given if more details are provided.