I have a Spring Service that is going through multiple items in a list and for each one it is making an extra WS call to external services. The Service is called by a Job on a fixed time interval.
As a first step, the Service is saving in a JOB_CONTROL table the status of the Job (STARTED), then it iterates through the list and at the end it saves it to (FINISHED).
There are 2 issues:
the JOB_CONTROL table doesn't get saved gradually - only the
"FINISHED" value is saved and never "STARTED"
if using flush method in order to force the commit, the table gets locked, eg. no other select can be made on it until the Service finishes
#Service
public class PromotionSchedulerService implements Runnable {
#Autowired
GeofencingAreaDAO storeDao;
#Autowired
promotionsWSClient promotionsWSClient;
#Autowired
private JobControlDAO jobControlDAO;
public void run() {
JobControl job = jobControlDAO.findByClassName(this.getClass().getSimpleName());
job.setState(JobControlStateTypes.RUNNING.getStateType());
job.setLastRunDate(new Date());
// LINE BELLOW DOES NOT GET COMMITED IN DB
jobControlDAO.save(job);
List < GeofencingArea > stores = storeDao.findAllStores();
for (GeofencingArea store: stores) {
/** Call WS **/
GetActivePromotionsResponse rsp = null;
try {
rsp = promotionsWSClient.getpromotions();
} catch (Exception e) {
e.printStackTrace();
job.setState(JobControlStateTypes.FAILED.getStateType());
job.setLastRunStatus("There was an error calling promagic promotions");
jobControlDAO.save(job);
return;
}
List < PromotionBean > promos = rsp.getReturn();
for (PromotionBean promo: promos) {
BackendPromotionPOJO backendPromotionsPOJO = new BackendPromotionPOJO();
backendPromotionsPOJO.setDescription(promo.getDescription());
}
}
// ONLY THIS JOB STATE GOES TO DB. IT ACTUALLY SEEM TO OVERWRITE PREVIOUS SET VALUE ("RUNNING") from line 16
job.setLastRunStatus("COMPLETED");
job.setState(JobControlStateTypes.SUCCESS.getStateType());
jobControlDAO.save(job);
}
}
I would like to force the commit after changing job state and not locking the table when doing this.
Related
I'm completely new to reactor programming and I'm really struggling with migrating old integration tests since upgrading to the latest Spring Boot / State Machine.
Most Integration tests have the same basic steps :
Call a method that returns a Mono and starts a state Machine and returns an object containing a generated unique id as well as some other infos related to the initial request.
With the returned object call a method that verifies if a value has been updated in the database (using the information of the object retried in step 1)
Poll at a fixed interval the method that checks in the database if value has changed until either the value has changed or a predefined timeout occurs.
Check another table in the database if another object has been updated
Below an example:
#Test
void testEndToEnd() {
var instance = ServiceInstance.buildDefault();
var updateRequest = UpdateRequest.build(instance);
// retrieve an update Response related to the request
// since a unique id is generated when triggering the update request
// before starting a stateMachine that goes through different steps
var updateResponse = service.updateInstance(updateRequest).block();
await().alias("Check if operation was successful")
.atMost(Duration.ofSeconds(120))
.pollInterval(Duration.ofSeconds(2))
.until(() -> expectOperationState(updateResponse, OperationState.SUCCESS))
// check if values are updated in secondary table
assertValuesInTransaction(updateResponse);
}
This was working fine before but ever since the latest update where it fails with the exception :
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread parallel-6
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:83)
at reactor.core.publisher.Mono.block(Mono.java:1710)
I saw that a good practice to test reactor methods using StepVerifier but I do not see how I can reproduce the part done with Awaitability to poll to see if the value has changed in the DB since the method that checks in the DB returns a Mono and not a flux that keeps sending values.
Any idea on how to accomplish this or to make the spring stack accept blocking operations?
Thanks
My current stack :
Spring Boot 3.0.1
Spring State Machine 3.0.1
Spring 6
Junit 5.9.2
So as discussed in comments here is an example with comments. I used flatMap to subscribe to what expectOperationState returns. Also there is Mono.fromCallable used which check the value from some method and if it fails to emit anything in 3 seconds - the timeout exception is thrown. Also we could try to get rid of this boolean value from expectOperationState and refactor the code to just return Mono<Void> with completed signal but this basically shows how you can achieve what you want.
class TestStateMachine {
#Test
void testUntilSomeOperationCompletes() {
final Service service = new Service();
final UpdateRequest updateRequest = new UpdateRequest();
StepVerifier.create(service.updateInstance(updateRequest)
.flatMap(updateResponse -> expectOperationState(updateResponse, OperationState.SUCCESS))
)
.consumeNextWith(Assertions::assertTrue)
.verifyComplete();
}
private Mono<Boolean> expectOperationState(final UpdateResponse updateResponse, final OperationState success) {
return Mono.fromCallable(() -> {
while (true) {
boolean isInDb = checkValueFromDb(updateResponse);
if (isInDb) {
return true;
}
}
})
.publishOn(Schedulers.single())
//timeout if we not receive any value from callable within 3 seconds so that we do not check forever
.timeout(Duration.ofSeconds(3));
}
private boolean checkValueFromDb(final UpdateResponse updateResponse) {
return true;
}
}
class Service {
Mono<UpdateResponse> updateInstance(final UpdateRequest updateRequest) {
return Mono.just(new UpdateResponse());
}
}
Here is an example without using Mono<Boolean> :
class TestStateMachine {
#Test
void test() {
final Service service = new Service();
final UpdateRequest updateRequest = new UpdateRequest();
StepVerifier.create(service.updateInstance(updateRequest)
.flatMap(updateResponse -> expectOperationState(updateResponse, OperationState.SUCCESS).timeout(Duration.ofSeconds(3)))
)
.verifyComplete();
}
private Mono<Void> expectOperationState(final UpdateResponse updateResponse, final OperationState success) {
return Mono.fromCallable(() -> {
while (true) {
boolean isInDb = checkValueFromDb(updateResponse);
if (isInDb) {
//return completed Mono
return Mono.<Void>empty();
}
}
})
.publishOn(Schedulers.single())
//timeout if we not receive any value from callable within 3 seconds so that we do not check forever
.timeout(Duration.ofSeconds(3))
.flatMap(objectMono -> objectMono);
}
private boolean checkValueFromDb(final UpdateResponse updateResponse) {
return true;
}
}
I want to save one document and one row in single transaction, but i want still be able to rollback transaction..
#Transactional(transactionManager = "chainedTransactionManager")
public void createAlienAndSpaceShip(String alienname, String spaceshipname){
spaceShipRepository.save(new SpaceShip(null, spaceshipname, 100.0d));
if (true) {
throw new RuntimeException("Something happened");
}
alienRepository.save(new Alien(null, alienname, 1.0d, 100.0d));
}
I tried to do this using ChainedTransactionManager, but it is deprecated..
I followed following tutorial https://www.youtube.com/watch?v=qOfdE-cFzto
I don't understand the behavior for distributed locks obtained from a JdbcLockRegistry.
#Bean
public LockRepository lockRepository(DataSource datasource) {
return new DefaultLockRepository(datasource);
}
#Bean
public LockRegistry lockRegistry(LockRepository repository) {
return new JdbcLockRegistry(repository);
}
My project is running upon PostgreSQL and Spring Boot version is 2.2.2
And this is the demonstration use case :
#GetMapping("/isolate")
public String isolate() throws InterruptedException {
Lock lock = registry.obtain("the-lock");
if (lock.tryLock(10, TimeUnit.SECONDS)) { // close
try {
Thread.sleep(30 * 1000L);
} finally {
lock.unlock(); // open
}
} else {
return "rejected";
}
return "acquired";
}
NB: that use case works when playing with Hazelcast distributed locks.
The observed behavior is that a first lock is duly registered in database through a call to the API on a first instance.
Then, within 30 seconds, a second on is requested on a different instance (other port), and it's updating the existing int_lock table's line (client_id changes) instead of failing. So the first endpoint delivers after 30 seconds (no unlock failure), and the second endpoint is delivering after its own period of 30 seconds. There is no mutual exclusion.
These are the logs for a single acquisition :
Trying to acquire lock...
Executing prepared SQL update
Executing prepared SQL statement [DELETE FROM INT_LOCK WHERE REGION=? AND LOCK_KEY=? AND CREATED_DATE<?]
Executing prepared SQL update
Executing prepared SQL statement [UPDATE INT_LOCK SET CREATED_DATE=? WHERE REGION=? AND LOCK_KEY=? AND CLIENT_ID=?]
Executing prepared SQL update
Executing prepared SQL statement [INSERT INTO INT_LOCK (REGION, LOCK_KEY, CLIENT_ID, CREATED_DATE) VALUES (?, ?, ?, ?)]
Processing...
Executing prepared SQL update
Executing prepared SQL statement [DELETE FROM INT_LOCK WHERE REGION=? AND LOCK_KEY=? AND CLIENT_ID=?]
It sounds strange that acquisition process begins with DELETE, though...
I've tried to set a constant client id for the DefaultLockRepository, without improvement.
Does anyone have a clue of understanding of how to fix this ? Thx for any help.
All right. It happens that the repository's TTL is 10s by default, just like my timeout in that specific use case. So the lock obviously dies (DELETE) before timeout period.
Here is a fix then:
#Bean
public LockRepository lockRepository(DataSource datasource) {
DefaultLockRepository repository = new DefaultLockRepository(datasource);
repository.setTimeToLive(60 * 1000);
return repository;
}
try lock.renew to extend lock period. lock.lock() doesn't update lock until it expires.
Trying to maintain a lock, I tried to take benefit of DefaultLockRepository#acquire, called by Lock#lock, which attempts update before inserting a new lock (and after cleaning up expired locks, as said before):
#GetMapping("/isolate")
public String isolate() throws InterruptedException {
Lock lock = registry.obtain("the-lock");
log.warn("Trying to acquire lock...");
if (lock.tryLock(10, TimeUnit.SECONDS)) { // close lock
try {
for (int i=0; i < 6; i++) { // very...
log.warn("Processing...");
Thread.sleep(5 * 1000L); // ... long task
lock.lock(); //DEBUG holding (lock update)
}
} finally {
if (!repository.isAcquired("the-lock")) {
throw new IllegalStateException("lock lost");
} else {
lock.unlock(); // open lock
}
}
} else {
return "rejected";
}
return "acquired";
}
But this didn't work as expected (NB: ttl is on default 10s in this test);
I always get a lock lost IllegalStateException in the end, despite the fact that I can see the lock date changing in PostgreSQL's console.
I am creating a e-prescribing module will allow several orders to be created and added to a list before being confirmed (saved in DB at once together). I have two forms - One where the order is created and the second with a link to add more orders, where the submit button will store all the orders (ArrayList items) in the DB. i.e. Each order upon creation gets added to an ArrayList [createNewDrugOrder() method below]
When I try to save each order inside this method - it works fine. but my requirement is to have the facility to store the orders as a draft before confirming them (storing in DB).
public class DrugordersPageController{
SessionFactory sessionfactory;
public void controller(){
if ("addDraftOrder".equals(action)){
createNewDrugOrder();
}
if ("confirmOrder".equals(action)){
saveOrder();
}
}
createNewDrugOrder(){
DrugOrder order = new DrugOrder();
order.setDose(1);
order.setDuration(1);
order.setQuantity(1);
ConfirmOrderClass.orderToConfirm.add(order);
}
saveOrder(){
DrugOrder order = ConfirmOrderClass.getDrugOrderMain();
order = (DrugOrder) Context.getOrderService().saveOrder(order, null); //Here is where error is thrown
}
}
class ConfirmOrderClass{
public static ArrayList<DrugOrder> orderToConfirm = new ArrayList<DrugOrder>();
public static ArrayList<DrugOrder> getDrugOrderMain(){
return orderToConfirm;
}
}
I understand this is a case of object being detached from the session, but I am unable to get it fixed. Calling session.save or session.update in the second form doesn't help..Neither does Hibernate.initialize(object) or adding a #Transactional notation
when I call Session s = sessionfactory.getCurrentSession or openSession, i still get null pointer error. Although I am able to retrieve the values of order.getDose(), order.getDuration() etc..
Please help!
Boot Pros,
I recently started to program in spring-boot and I stumbled upon a question where I would like to get your opinion on.
What I try to achieve:
I created a Controller that exposes a GET endpoint, named nonBlockingEndpoint. This nonBlockingEndpoint executes a pretty long operation that is resource heavy and can run between 20 and 40 seconds.(in the attached code, it is mocked by a Thread.sleep())
Whenever the nonBlockingEndpoint is called, the spring application should register that call and immediatelly return an Operation ID to the caller.
The caller can then use this ID to query on another endpoint queryOpStatus the status of this operation. At the beginning it will be started, and once the controller is done serving the reuqest it will be to a code such as SERVICE_OK. The caller then knows that his request was successfully completed on the server.
The solution that I found:
I have the following controller (note that it is explicitely not tagged with #Async)
It uses an APIOperationsManager to register that a new operation was started
I use the CompletableFuture java construct to supply the long running code as a new asynch process by using CompletableFuture.supplyAsync(() -> {}
I immdiatelly return a response to the caller, telling that the operation is in progress
Once the Async Task has finished, i use cf.thenRun() to update the Operation status via the API Operations Manager
Here is the code:
#GetMapping(path="/nonBlockingEndpoint")
public #ResponseBody ResponseOperation nonBlocking() {
// Register a new operation
APIOperationsManager apiOpsManager = APIOperationsManager.getInstance();
final int operationID = apiOpsManager.registerNewOperation(Constants.OpStatus.PROCESSING);
ResponseOperation response = new ResponseOperation();
response.setMessage("Triggered non-blocking call, use the operation id to check status");
response.setOperationID(operationID);
response.setOpRes(Constants.OpStatus.PROCESSING);
CompletableFuture<Boolean> cf = CompletableFuture.supplyAsync(() -> {
try {
// Here we will
Thread.sleep(10000L);
} catch (InterruptedException e) {}
// whatever the return value was
return true;
});
cf.thenRun(() ->{
// We are done with the super long process, so update our Operations Manager
APIOperationsManager a = APIOperationsManager.getInstance();
boolean asyncSuccess = false;
try {asyncSuccess = cf.get();}
catch (Exception e) {}
if(true == asyncSuccess) {
a.updateOperationStatus(operationID, Constants.OpStatus.OK);
a.updateOperationMessage(operationID, "success: The long running process has finished and this is your result: SOME RESULT" );
}
else {
a.updateOperationStatus(operationID, Constants.OpStatus.INTERNAL_ERROR);
a.updateOperationMessage(operationID, "error: The long running process has failed.");
}
});
return response;
}
Here is also the APIOperationsManager.java for completness:
public class APIOperationsManager {
private static APIOperationsManager instance = null;
private Vector<Operation> operations;
private int currentOperationId;
private static final Logger log = LoggerFactory.getLogger(Application.class);
protected APIOperationsManager() {}
public static APIOperationsManager getInstance() {
if(instance == null) {
synchronized(APIOperationsManager.class) {
if(instance == null) {
instance = new APIOperationsManager();
instance.operations = new Vector<Operation>();
instance.currentOperationId = 1;
}
}
}
return instance;
}
public synchronized int registerNewOperation(OpStatus status) {
cleanOperationsList();
currentOperationId = currentOperationId + 1;
Operation newOperation = new Operation(currentOperationId, status);
operations.add(newOperation);
log.info("Registered new Operation to watch: " + newOperation.toString());
return newOperation.getId();
}
public synchronized Operation getOperation(int id) {
for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
return op;
}
}
Operation notFound = new Operation(-1, OpStatus.INTERNAL_ERROR);
notFound.setCrated(null);
return notFound;
}
public synchronized void updateOperationStatus (int id, OpStatus newStatus) {
iteration : for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
op.setStatus(newStatus);
log.info("Updated Operation status: " + op.toString());
break iteration;
}
}
}
public synchronized void updateOperationMessage (int id, String message) {
iteration : for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if(op.getId() == id) {
op.setMessage(message);
log.info("Updated Operation status: " + op.toString());
break iteration;
}
}
}
private synchronized void cleanOperationsList() {
Date now = new Date();
for(Iterator<Operation> iterator = operations.iterator(); iterator.hasNext();) {
Operation op = iterator.next();
if((now.getTime() - op.getCrated().getTime()) >= Constants.MIN_HOLD_DURATION_OPERATIONS ) {
log.info("Removed operation from watchlist: " + op.toString());
iterator.remove();
}
}
}
}
The questions that I have
Is that concept a valid one that also scales? What could be improved?
Will i run into concurrency issues / race conditions?
Is there a better way to achieve the same in boot spring, but I just didn't find that yet? (maybe with the #Async directive?)
I would be very happy to get your feedback.
Thank you so much,
Peter P
It is a valid pattern to submit a long running task with one request, returning an id that allows the client to ask for the result later.
But there are some things I would suggest to reconsider :
do not use an Integer as id, as it allows an attacker to guess ids and to get the results for those ids. Instead use a random UUID.
if you need to restart your application, all ids and their results will be lost. You should persist them to a database.
Your solution will not work in a cluster with many instances of your application, as each instance would only know its 'own' ids and results. This could also be solved by persisting them to a database or Reddis store.
The way you are using CompletableFuture gives you no control over the number of threads used for the asynchronous operation. It is possible to do this with standard Java, but I would suggest to use Spring to configure the thread pool
Annotating the controller method with #Async is not an option, this does not work no way. Instead put all asynchronous operations into a simple service and annotate this with #Async. This has some advantages :
You can use this service also synchronously, which makes testing a lot easier
You can configure the thread pool with Spring
The /nonBlockingEndpoint should not return the id, but a complete link to the queryOpStatus, including id. The client than can directly use this link without any additional information.
Additionally there are some low level implementation issues which you may also want to change :
Do not use Vector, it synchronizes on every operation. Use a List instead. Iterating over a List is also much easier, you can use for-loops or streams.
If you need to lookup a value, do not iterate over a Vector or List, use a Map instead.
APIOperationsManager is a singleton. That makes no sense in a Spring application. Make it a normal PoJo and create a bean of it, get it autowired into the controller. Spring beans by default are singletons.
You should avoid to do complicated operations in a controller method. Instead move anything into a service (which may be annotated with #Async). This makes testing easier, as you can test this service without a web context
Hope this helps.
Do I need to make database access transactional ?
As long as you write/update only one row, there is no need to make this transactional as this is indeed 'atomic'.
If you write/update many rows at once you should make it transactional to guarantee, that either all rows are updated or none.
However, if two operations (may be from two clients) update the same row, always the last one will win.