In my current web application, i use #RestController with CompletableFuture result for all services.
Database operations are asynchronous (CompletableFuture methods), but i would like commit operations only before send result
i would like to commit database modifications after --save-- asynchronous ended (--save-- is a list of future business)
#RestController
public class MyController {
...
#RequestMappping(...)
public CompletableFuture<ResponseEntity<AnyResource>> service(...){
CompletableFuture ...
.thenCompose(--check--)
.thenAsync(--save--)
...ect
.thenApply(
return ResponseEntity.ok().body(theResource);
);
}
}
-> i've tried with #Transactional, but it doesn't work (commit at the method's end but async method partially or not executed
-> Other way with programmatic :
#RequestMappping(...)
public CompletableFuture<ResponseEntity<AnyResource>> service(...){
DefaultTransactionDefinition def = new DefaultTransactionDefinition();
// explicitly setting the transaction name is something that can only be done programmatically
def.setName("SomeTxName");
def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRED);
TransactionStatus status = this.platformTransactionManager.getTransaction(def);
CompletableFuture ...
.thenCompose(--check--)
.thenAsync(--save--)
...ect
.thenApply(
this.platformTransactionManager.commit(status)
return ResponseEntity.ok().body(theResource);
);
}
An error occured "Cannot deactivate transaction synchronization - not active", supposed because not the same thread.
Is there a proper way, to use transactional with CompletableFuture ?
Related
Working in a Spring Boot context, I am testing a service that queries a db, then makes a remote call that updates the db, then requeries the db to get the updated data.
I am trying to use Wiremock or MockServer to mock the remote call, but can't figure out how to update the embedded database as part of the mock server response generation process.
For example, using MockServer, I tried creating a callback class that had an autowired JdbcTemplate instance, but in the context of the callback that JdbcTemplate variable is null.
public class ApprovalHappyPathCallback implements ExpectationResponseCallback {
JdbcTemplate jdbcTemplate;
#Autowired
public void setDataSource(DataSource ds) {
jdbcTemplate = new JdbcTemplate(ds);
}
public static final HttpResponse httpResponse = response()
.withStatusCode(HttpStatusCode.ACCEPTED_202.code())
.withHeaders(
header("x-callback", "test_callback_header"),
header("Content-Length", "a_callback_response".getBytes(UTF_8).length),
header("Connection", "keep-alive")
)
.withBody("a_callback_response");
#Override
public HttpResponse handle(HttpRequest httpRequest) {
if (httpRequest.getMethod().equals("GET")) {
jdbcTemplate.execute("update communications set status = 'APPROVED_SCHEDULED' where id = 153511");
return httpResponse;
} else {
return notFoundResponse();
}
}
}
The call back executes, but the jdbcTemplate statement does not work.
The callback is referenced like this in the test:
mockServer.when(request().withMethod("GET"))
.withBody("Approved")
// );
.respond(
callback()
.withCallbackClass(ApprovalHappyPathCallback.class)
);
The service method that makes the remote call is essentially:
public CommunicationEntity approveCommunication(Long communicationId) {
String approvalToken = commRepo.approvalTokenById(communicationId);
if (approvalToken == null) {
approvalToken = UUID.randomUUID().toString();
communicationEntity.setApprovalToken(approvalToken);
commRepo.save(communicationEntity);
}
String approvalResponse = remoteCommunicationApprover.approveCommunication(communicationId, approvalToken);
CommunicationEntity communicationEntity = getCommunicationById(communicationId);
if (communicationEntity.getStatus() != CommunicationStatus.Approved_Scheduled) {
throw new BadRequestException(
"Approval request for communication " + communicationId + " and token " + approvalToken
+ " failed with remote response: " + approvalResponse,
ErrorCodes.COMMUNICATION_SVC_REMOTE_APPROVAL_REQUEST_FAILED);
}
return communicationEntity;
There were two issues causing problems: making sure the jdbcTemplate used in the callback method was configured with the correct DataSource, and making sure that the data in the embedded in memory DB was accessible from the MockServer response generation thread.
I solved the first problem by using a lambda or closure for the MockServer callback in which I use the JdbcTemplate instance created in the test class with the autowired DataSource (though solutions exist for the callback class approach as well).
The second problem was the result of the fact that the test method was within a transaction and so inserts to the DB made at the beginning of the test were not committed when the MockServer thread (note that the MockServer response generation happens in a different thread than the main thread where the test method is running) was executing the callback. Thus those inserts were not accessible to the callback.
The solution was to annotate the test method with #Transactional(propagation = Propagation.NOT_SUPPORTED)
See h2 database access to test data from separate threads
Background of the code:
In order to replicate a production scenario, I have created a dummy app that will basically save something in DB in a transaction, and in the same method, it publishEvent and publishEvent send a message to rabbitMQ.
Classes and usages
Transaction Starts from this method.:
#Override
#Transactional
public EmpDTO createEmployeeInTrans(EmpDTO empDto) {
return createEmployee(empDto);
}
This method saves the record in DB and also triggers publishEvent
#Override
public EmpDTO createEmployee(EmpDTO empDTO) {
EmpEntity empEntity = new EmpEntity();
BeanUtils.copyProperties(empDTO, empEntity);
System.out.println("<< In Transaction : "+TransactionSynchronizationManager.getCurrentTransactionName()+" >> Saving data for employee " + empDTO.getEmpCode());
// Record data into a database
empEntity = empRepository.save(empEntity);
// Sending event , this will send the message.
eventPublisher.publishEvent(new ActivityEvent(empDTO));
return createResponse(empDTO, empEntity);
}
This is ActivityEvent
import org.springframework.context.ApplicationEvent;
import com.kuldeep.rabbitMQProducer.dto.EmpDTO;
public class ActivityEvent extends ApplicationEvent {
public ActivityEvent(EmpDTO source) {
super(source);
}
}
And this is TransactionalEventListener for the above Event.
//#Transactional(propagation = Propagation.REQUIRES_NEW)
#TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void onActivitySave(ActivityEvent activityEvent) {
System.out.println("Activity got event ... Sending message .. ");
kRabbitTemplate.convertAndSend(exchange, routingkey, empDTO);
}
This is kRabbitTemplate is a bean config like this :
#Bean
public RabbitTemplate kRabbitTemplate(ConnectionFactory connectionFactory) {
final RabbitTemplate kRabbitTemplate = new RabbitTemplate(connectionFactory);
kRabbitTemplate.setChannelTransacted(true);
kRabbitTemplate.setMessageConverter(kJsonMessageConverter());
return kRabbitTemplate;
}
Problem Definition
When I am saving a record and sending a message on rabbitMQ using the above code flow, My messages are not delivered on the server means they lost.
What I understand about the transaction in AMQP is :
If the template is transacted, but convertAndSend is not called from Spring/JPA Transaction then messages are committed within the template's convertAndSend method.
// this is a snippet from org.springframework.amqp.rabbit.core.RabbitTemplate.doSend()
if (isChannelLocallyTransacted(channel)) {
// Transacted channel created by this template -> commit.
RabbitUtils.commitIfNecessary(channel);
}
But if the template is transacted and convertAndSend is called from Spring/JPA Transaction then this isChannelLocallyTransacted in doSend method will evaluate false and commit will be done in the method which initiated Spring/JPA Transaction.
What I found after investigating the reason for message loss in my above code.
Spring transaction was active when I called convertAndSend method, so it was supposed to commit the message in Spring transaction.
For that, RabbitTemplate binds the resources and registers the Synchronizations before sending the message in bindResourceToTransaction of org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.
public static RabbitResourceHolder bindResourceToTransaction(RabbitResourceHolder resourceHolder,
ConnectionFactory connectionFactory, boolean synched) {
if (TransactionSynchronizationManager.hasResource(connectionFactory)
|| !TransactionSynchronizationManager.isActualTransactionActive() || !synched) {
return (RabbitResourceHolder) TransactionSynchronizationManager.getResource(connectionFactory); // NOSONAR never null
}
TransactionSynchronizationManager.bindResource(connectionFactory, resourceHolder);
resourceHolder.setSynchronizedWithTransaction(true);
if (TransactionSynchronizationManager.isSynchronizationActive()) {
TransactionSynchronizationManager.registerSynchronization(new RabbitResourceSynchronization(resourceHolder,
connectionFactory));
}
return resourceHolder;
}
In my code, after resource bind, it is not able to registerSynchronization because TransactionSynchronizationManager.isSynchronizationActive()==false. and since it fails to registerSynchronization, spring commit did not happen for the rabbitMQ message as AbstractPlatformTransactionManager.triggerAfterCompletion calls RabbitMQ's commit for each synchronization.
What problem I faced because of the above issue.
Message was not committed in the spring transaction, so the message lost.
As resource was added in bindResourceToTransaction, this resource remained bind and did not let add the resource for any other message to send in the same thread.
Possible Root Cause of TransactionSynchronizationManager.isSynchronizationActive()==false
I found the method which starts the transaction removed the synchronization in triggerAfterCompletion of org.springframework.transaction.support.AbstractPlatformTransactionManager class. because status.isNewSynchronization() evaluated true after DB opertation (this usually not happens if I call convertAndSend without ApplicationEvent).
private void triggerAfterCompletion(DefaultTransactionStatus status, int completionStatus) {
if (status.isNewSynchronization()) {
List<TransactionSynchronization> synchronizations = TransactionSynchronizationManager.getSynchronizations();
TransactionSynchronizationManager.clearSynchronization();
if (!status.hasTransaction() || status.isNewTransaction()) {
if (status.isDebug()) {
logger.trace("Triggering afterCompletion synchronization");
}
// No transaction or new transaction for the current scope ->
// invoke the afterCompletion callbacks immediately
invokeAfterCompletion(synchronizations, completionStatus);
}
else if (!synchronizations.isEmpty()) {
// Existing transaction that we participate in, controlled outside
// of the scope of this Spring transaction manager -> try to register
// an afterCompletion callback with the existing (JTA) transaction.
registerAfterCompletionWithExistingTransaction(status.getTransaction(), synchronizations);
}
}
}
What I Did to overcome on this issue
I simply added #Transactional(propagation = Propagation.REQUIRES_NEW) along with on #TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT) in onActivitySave method and it worked as a new transaction was started.
What I need to know
Why this status.isNewSynchronization in triggerAfterCompletion method when using ApplicationEvent?
If the transaction was supposed to terminate in the parent method, why I got TransactionSynchronizationManager.isActualTransactionActive()==true in Listner class?
If Actual Transaction Active, was it supposed to remove the synchronization?
In bindResourceToTransaction, do spring AMQP assumed an active transaction without synchronization? if the answer is yes, why not to synchronization. init if it is not activated?
If I am propagating a new transaction then I am losing the parent transaction, is there any better way to do it?
Please help me on this, it is a hot production issue, and I am not very sure about the fix I have done.
This is a bug; the RabbitMQ transaction code pre-dated the #TransactionalEventListener code, by many years.
The problem is, with this configuration, we are in a quasi-transactional state, while there is indeed a transaction in process, the synchronizations are already cleared because the transaction has already committed.
Using #TransactionalEventListener(phase = TransactionPhase.BEFORE_COMMIT) works.
I see you already raised an issue:
https://github.com/spring-projects/spring-amqp/issues/1309
In future, it's best to ask questions here, or raise an issue if you feel there is a bug. Don't do both.
Let's say I have a rest call which creates some object "A" in the database. Method is marked with #Transactional annotation. And just after creation I need to launch another asynchronous process in another thread or through some messaging system or in some other async way. That new process depends on the object "A" and needs to see it.
How can I make sure that transaction is commited before new process starts execution?
For example in Spring there is
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization(){
void afterCommit(){
//do what you want to do after commit
}
})
Does Quarkus has something similar?
You can inject TransactionManager
#Inject
TransactionManager transactionManager;
and do
Transaction transaction = transactionManager.getTransaction();
transaction.registerSynchronization(new Synchronization() {
#Override
public void beforeCompletion() {
//nothing here
}
#Override
public void afterCompletion(int status) {
//do some code after completion
}
});
Complete structure of the program
Annotation:
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.METHOD)
public #interface UserAnnotation {
}
Then created a Interceptor:
public class UserInterceptor implements MethodInterceptor {
private static final Logger logger = LoggerFactory.getLogger(UserInterceptor.class);
#Inject
UserService userService; // this is not working
public Object invoke(MethodInvocation invocation) throws Throwable {
logger.info("UserInterceptor : Interceptor Invoked");
Object result = invocation.proceed();
Observable<List<User>> observable = (Observable<List<Sample>>) result;
observable.flatMap(Observable::from).subscribe(object -> {
User user = (User)object
SampleSender sender = new SampleSender();
sender.setBoolean(user.isBoolean());
logger.info("Pushing Data into Sender");
userService.insert(String.join("_", "key", "value"), sender);
}
return result;
}
}
Then I created a GuiceModule as below:-
public class UserModule extends AbstractModule {
#Override
protected void configure() {
SampleInterceptor interceptor = new SampleInterceptor()
requestInjection(interceptor);
bindInterceptor(Matchers.any(), Matchers.annotatedWith(SampleAnnotation.class), interceptor);
}
}
Class in which I am using the above annotation is
// This class also have so many method and this was already declared and using in another services, I created a sample class here
class UserClassForInterceptor {
#Inject
AnotherClass anotherClass;
// this userMethod() is not a new method, its already created,
// now I am adding annotation to it, because after finishing this functionality,
// I want something should be done, so created annotation and added here
#UserAnnotation
public Observable<List<Sample>> userMethod() {
logger.info("This is printing only once");
return anotherClass.getUser().flatMap(user ->{
logger.info("This is also printing twice");
// this logger printed twise means, this code snippet is getting executed twise
});
}
}
public class AnotherClass{
public Observable<User> getUser(){
Observable<Sample> observableSample = methodReturnsObservableSample();
logger.info("Getting this logger only once");
return observableSample.map(response-> {
logger.info("This logger is printing twice");
//here have code to return observable of User
});
}
}
If I remove annotation loggers inside the observable are printing only one time but when I use annotation those loggers are getting printed twise. Why it is behaving like this I dont know.
I have a RestModule using which I am binding UserClassForInterceptor as follows
public final class RestModule extends JerseyServletModule {
// other classes binding
bind(UserClassForInterceptor.class).in(Scopes.SINGLETON);
// other classes binding
install(new SampleModule());
}
Now I have a bootsrap class in which I am binding RestModule
public class Bootstrap extends ServerBootstrap {
binder.install(new RestModule());
}
Usage:-
#Path("service/sample")
public class SampleRS {
#Inject
UserClassForInterceptor userClassForInterceptor;
public void someMethod() {
userClassForInterceptor.sampleMethod();
}
}
You created an annotation, #UserAnnotation, and an interceptor class to go with the annotation. You attach the annotation to a method, userMethod().
The first thing your interceptor routine does is invoke userMethod() to get the observable that it returns and then the interceptor subscribes to the returned observable, causing the first log messages to appear. Eventually, the interceptor returns the observable to the original caller. When something else subscribes to the returned observable, the observer chain is activated a second time, hence the log messages appear twice.
RxJava Has Side Effects
While RxJava is an implementation of the "functional reactive programming" concept, the observer chains that you construct (in a functional manner) only work when they are subscribed to, and those subscriptions have side effects. Logging output is one side effect, and probably the most benign; changes to variables or invocations of methods that have side effects have a wider impact.
When an observer chain is constructed (properly), it acts as a potential computation until there is a subscriber. If you need to have more than one subscriber, as you might for your problem domain, then you have to decide whether the observer chain needs to be activated for each subscription, the normal case, or only once for all overlapping subscriptions.
If you want all overlapping subscriptions to share the same observable, then you can use the share() operator. There are a number of related operators that affect the lifetime of observables and subscriptions. Here is an overview: How to use RxJava share() operator?
Aspect Oriented Programming: Interceptors And Guice
Your code is using Guice to provide a capability called "aspect oriented programming". This allows you to introduce code into your program to address cross-cutting concerns, or to enhance its functionality by setting up controlled gateways. Using Guice, or similar AOP approaches, requires discipline.
In your case, you used the interception process to cause unexplained (until now) side effects by subscribing to an observer chain that has non-trivial side effects. Imagine that the method you intercepted set up a one-time connection and that your interceptor used up that connection doing its work, leaving the original caller unable to use the connection.
The discipline you need is to understand the rules that the interceptor must follow. Think of rules such as "First, do no harm".
Doing Things The FRP Way
If you need to add an extra step when handling user information, then you should construct a new observable in your interceptor that does that, but only when the original caller subscribed to the observable:
Object result = invocation.proceed();
Observable<List<User>> observable = (Observable<List<Sample>>) result;
Observable<List<User>> newObservable = observable
.doOnNext( sampleList ->
Observable.fromIterable( sampleList )
.subscribe(object -> {
User user = (User)object
SampleSender sender = new SampleSender();
sender.setBoolean(user.isBoolean());
logger.info("Pushing Data into Sender");
userService.insert(String.join("_", "key", "value"), sender);
}));
return newObservable;
By returning a modified observer chain, you don't introduce side effects from the original observer chain, and ensure that the side effects you introduce in your own code will only be triggered when the original observer chain is subscribed to.
This code also helped me
public Object invoke(MethodInvocation invocation) throws Throwable {
Object result = null;
try{
logger.debug("Interceptor Invoked");
result = invocation.proceed();
Observable<List<User>> observable = (Observable<List<User>>)result;
return observable
.doOnNext(this::updateUser);
}
catch(Exception ex){
logger.error("Error: ",ex);
}
return result;
}
private void updateUser(List<User> users) {
if(CollectionUtils.isNotEmpty(users)) {
for(User user: users) {
SampleSender sender = new SampleSender();
sender.setBoolean(user.isBoolean());
logger.info("Pushing Data into Sender");
userService.insert(String.join("_", "key", "value"), sender);
}
}
}
I have implemented a simple Rest service by which I'd like to test deferredResult from Spring. While am I getting texts in that order:
TEST
TEST 1
TEST AFTER DEFERRED RESULT
I am very interested why in a browser (client) I need to wait that 8 seconds. Isn't that deferedResult shouldn't be non-blocking and run a task in the background? If no, how to create a rest service which will be non-blocking and run tasks in the background without using Java 9 and reactive streams?
#RestController("/")
public class Controller {
#GetMapping
public DeferredResult<Person> test() {
System.out.println("TEST");
DeferredResult<Person> result = new DeferredResult<>();
CompletableFuture.supplyAsync(this::test1)
.whenCompleteAsync((res, throwable) -> {
System.out.println("TEST AFTER DEFERRED RESULT");
result.setResult(res);
});
System.out.println("TEST 1");
return result;
}
private Person test1() {
try {
Thread.sleep(8000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return new Person("michal", 20);
}
}
class Person implements Serializable {
private String name;
private int age;
}
DeferredResult is a holder for a WebRequest to allow the serving thread to release and serve another incoming HTTP request instead of waiting for the current one's result. After setResult or setError methods will be invoked - Spring will release that stored WebRequest and your client will receive the response.
DeferredResult holder is a Spring Framework abstraction for Non-blocking IO threading.
Deferred result abstraction has nothing with background tasks. Calling it without threading abstractions will cause the expected same thread execution. Your test1 method is running in the background because of CompletableFuture.supplyAsync method invocation that gives the execution to common pool.
The result is returned in 8 seconds because the whenCompleteAsync passed callback will be called only after test1 method will return.
You cannot receive the result immediately when your "service call logic" takes 8 seconds despite you are performing it in the background. If you want to release the HTTP request - just return an available proper object (it could contain a UUID, for example, to fetch the created person later) or nothing from the controller method. You can try to GET your created user after N seconds. There are specific HTTP response codes (202 ACCEPTED), that means the serverside is processing the request. Finally just GET your created object.
The second approach (if you should notify your clientside - but I will not recommend you to do it if this is the only reason) - you can use WebSockets to notify the clientside and message with it.