From time to time Spring REST function fails with: "kotlinx.coroutines.JobCancellationException: MonoCoroutine was cancelled".
It is suspend function which calls another service using spring-webflux client. There are multiple suspend functions in my rest class. Looks like this problem occurs when multiple requests arrive to the same time. But may be not :-)
Application runs on Netty server.
Example:
#GetMapping("/customer/{id}")
suspend fun getCustomer(#PathVariable #NotBlank id: String): ResponseEntity<CustomerResponse> =
withContext(MDCContext()) {
ResponseEntity.status(HttpStatus.OK)
.body(customerService.aggregateCustomer(id))
}
Service call:
suspend fun executeServiceCall(vararg urlData: Input) = webClient
.get()
.uri(properties.url, *urlData)
.retrieve()
.bodyToMono(responseTypeRef)
.retryWhen(
Retry.fixedDelay(properties.retryCount, properties.retryBackoff)
.onRetryExhaustedThrow { _, retrySignal ->
handleRetryException(retrySignal)
}
.filter { it is ReadTimeoutException || it is ConnectTimeoutException }
)
.onErrorMap {
// throw exception
}
.awaitFirstOrNull()
Part of Stack Trace:
Caused by: kotlinx.coroutines.JobCancellationException: MonoCoroutine was cancelled; job="coroutine#1":MonoCoroutine{Cancelling}#650774ce
at kotlinx.coroutines.JobSupport.cancel(JobSupport.kt:1578)
at kotlinx.coroutines.Job$DefaultImpls.cancel$default(Job.kt:183)
at kotlinx.coroutines.reactor.MonoCoroutine.dispose(Mono.kt:122)
at reactor.core.publisher.FluxCreate$SinkDisposable.dispose(FluxCreate.java:1033)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.disposeResource(MonoCreate.java:313)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.cancel(MonoCreate.java:300)
Related
I am struggling with proper coroutine usage on cache handling using spring boot #Cacheable with ehcache on two methods:
calling another service using webclient:
suspend fun getDeviceOwner(correlationId: String, ownerId: String): DeviceOwner{
webClient
.get()
.uri(uriProvider.provideUrl())
.header(CORRELATION_ID, correlationId)
.retrieve()
.onStatus(HttpStatus::isError) {response ->
Mono.error(
ServiceCallExcpetion("Call failed with: ${response.statusCode()}")
)
}.awaitBodyOrNull()
?: throw ServiceCallExcpetion("Call failed with - response is null.")
}
calling db using r2dbc
suspend fun findDeviceTokens(ownerId: UUID, deviceType: String) {
//CoroutineCrudRepository.findTokens
}
What seems to be working for me is calling from:
suspend fun findTokens(data: Data): Collection<String> = coroutineScope {
val ownership = async(Dispatchers.IO, CoroutineStart.LAZY) { service.getDeviceOwner(data.nonce, data.ownerId) }.await()
val tokens = async(Dispatchers.IO, CoroutineStart.LAZY) {service.findDeviceTokens(ownership.ownerId, ownership.ownershipType)}
tokens.await()
}
#Cacheable(value = ["ownerCache"], key = "#ownerId")
fun getDeviceOwner(correlationId: String, ownerId: String)= runBlocking(Dispatchers.IO) {
//webClientCall
}
#Cacheable("deviceCache")
override fun findDeviceTokens(ownerId: UUID, deviceType: String) = runBlocking(Dispatchers.IO) {
//CoroutineCrudRepository.findTokens
}
But from what I am reading it's not good practise to use runBlocking.
https://kotlinlang.org/docs/coroutines-basics.html#your-first-coroutine
Would it block the main thread or the thread which was designated by the parent coroutine?
I also tried with
#Cacheable(value = ["ownerCache"], key = "#ownerId")
fun getDeviceOwnerAsync(correlationId: String, ownerId: String) = GlobalScope.async(Dispatchers.IO, CoroutineStart.LAZY) {
//webClientCall
}
#Cacheable("deviceCache")
override fun findDeviceTokensAsync(ownerId: UUID, deviceType: String) = GlobalScope.async(Dispatchers.IO, CoroutineStart.LAZY) {
//CoroutineCrudRepository.findTokens
}
Both called from suspended function without any additional coroutineScope {} and async{}
suspend fun findTokens(data: Data): Collection<String> =
service.getDeviceOwnerAsync(data.nonce,data.ownerId).await()
.let{service.findDeviceTokensAsync(it.ownerId, it.ownershipType).await()}
I am reading that using GlobalScope is not good practise either due to possible endless run of this coroutine when something stuck or long response (in very simple words). Also in this approach, using GlobalScope, when I tested negative scenarios and external ms call resulted with 404(on purpose) result was not stored in the cache (as I excepted) but for failing CoroutineCrudRepository.findTokens call (throwing exception) Deferred value was cached which is not what I wanted. Storing failing exececution results is not a thing with runBlocking.
I tried also #Cacheable("deviceCache", unless = "#result.isCompleted == true && #result.isCancelled == true")
but it also seems to not work as I would imagine.
Could you please advice the best coroutine approach with correct exception handling for integrating with spring boot caching which will store value in cache only on non failing call?
Although annotations from Spring Cache abstraction are fancy, I also, unfortunately, haven't found any official solution for using them side by side with Kotlin coroutines.
Yet there is a library called spring-kotlin-coroutine that claims to solve this issue. Though, never tried as it doesn't seem to be maintained any longer - the last commit was pushed in May 2019.
For the moment I've been using CacheManager bean and managing the aforementioned manually. I found that a better solution rather than blocking threads.
Sample code with Redis as a cache provider:
Dependency in build.gradle.kts:
implementation("org.springframework.boot:spring-boot-starter-data-redis-reactive")
application.yml configuration:
spring:
redis:
host: redis
port: 6379
password: changeit
cache:
type: REDIS
cache-names:
- account-exists
redis:
time-to-live: 3m
Code:
#Service
class AccountService(
private val accountServiceApiClient: AccountServiceApiClient,
private val redisCacheManager: RedisCacheManager
) {
suspend fun isAccountExisting(accountId: UUID): Boolean {
if (getAccountExistsCache().get(accountId)?.get() as Boolean? == true) {
return true
}
val account = accountServiceApiClient.getAccountById(accountId) // this call is reactive
if (account != null) {
getAccountExistsCache().put(account.id, true)
return true
}
return false
}
private fun getAccountExistsCache() = redisCacheManager.getCache("account-exists") as RedisCache
}
In the Kotlin Coroutines context, every suspend function has 1 additional param of type kotlin.coroutines.Continuation<T>, that's why the org.springframework.cache.interceptor.SimpleKeyGenerator generates always a wrong key. Also, the CacheInterceptor does not know anything about suspend functions, so, it stores a COROUTINE_SUSPENDED object instead of the actual value, without evaluating the suspended wrapper.
You can check this repository https://github.com/konrad-kaminski/spring-kotlin-coroutine, they added Cache support for Coroutines, the specific Cache support implementation is here -> https://github.com/konrad-kaminski/spring-kotlin-coroutine/blob/master/spring-kotlin-coroutine/src/main/kotlin/org/springframework/kotlin/coroutine/cache/CoroutineCacheConfiguration.kt.
Take a look at CoroutineCacheInterceptor and CoroutineAwareSimpleKeyGenerator,
Hope this fixes your issue
I'm a new in Kotlin's coroutines.
Here code with classic Thread:
import com.google.gson.JsonElement
import com.google.gson.JsonObject
import com.google.gson.JsonParser
import com.zaxxer.hikari.HikariConfig
import com.zaxxer.hikari.HikariDataSource
import okhttp3.*
import okio.ByteString
import org.slf4j.LoggerFactory
import java.util.concurrent.atomic.AtomicInteger
object BithumbSocketListener : WebSocketListener() {
override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) {
super.onFailure(webSocket, t, response)
Thread {
оkHttpClient.newWebSocket(wsRequest, BithumbSocketListener)
}.start()
}
override fun onMessage(webSocket: WebSocket, text: String) {
super.onMessage(webSocket, text)
logger.debug("ws_onMessage: text = $text")
}
}
fun main(args: Array<String>) {
currenciesList = currencies.split(",")
currenciesList.forEach {
OkHttpClient().newWebSocket(wsRequest, BithumbSocketListener)
}
}
As you can see I have list of currencies (currenciesList). I iterate it and call newWebSocket for every item of list. As you can see BithumbSocketListener is a singleton.
If has some problem with web socket then call callback method onFailure and I create new web socket in separate java thread:
Thread {
оkHttpClient.newWebSocket(wsRequest, BithumbSocketListener)
}.start()
Nice. It's work fine.
But I want replace this code by Kotlin coroutines.
How I can do this?
Thanks.
Since you're processing an async stream of messages, you should port it to coroutines by implementing an actor, such as
val wsActor: SendChannel<String> = actor {
for (msg in channel) {
logger.info("Another message is in: ${msg}")
}
}
From the type of wsActor you can see you're supposed to send messages to it. This is where the bridging code comes in:
class BithumbSocketListener(
private val chan: Channel<String>
) : WebSocketListener() {
override fun onMessage(webSocket: WebSocket, text: String) {
chan.send(text)
}
override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) {
оkHttpClient.newWebSocket(wsRequest, this)
}
}
Note that, compared to your code, I don't start any new threads for retrying. This has nothing to do with porting to coroutines, your code doesn't need it either. newWebSocket is an async call that returns immediately.
Finally, start the websockets for each currency:
currenciesList.forEach {
OkHttpClient().newWebSocket(wsRequest, BithumbSocketListener(wsActor)
}
Using spring reactive WebClient, I consume an API and in case of response with 500 status I need to retry with exponential backoff. But in Mono class, I don't see any retryBackoff with Predicate as input parameter.
This is the kind of function I search for:
public final Mono<T> retryBackoff(Predicate<? super Throwable> retryMatcher, long numRetries, Duration firstBackoff)
Right now my implementation is as following (I don't have retry with backOff mechanism):
client.sendRequest()
.retry(e -> ((RestClientException) e).getStatus() == 500)
.subscribe();
You might want to have a look at the reactor-extra module in the reactor-addons project. In Maven you can do:
<dependency>
<groupId>io.projectreactor.addons</groupId>
<artifactId>reactor-extra</artifactId>
<version>3.2.3.RELEASE</version>
</dependency>
And then use it like this:
client.post()
.syncBody("test")
.retrieve()
.bodyToMono(String.class)
.retryWhen(Retry.onlyIf(ctx -> ctx.exception() instanceof RestClientException)
.exponentialBackoff(firstBackoff, maxBackoff)
.retryMax(maxRetries))
Retry.onlyIf is now deprecated/removed.
If anyone is interested in the up-to-date solution:
client.post()
.syncBody("test")
.retrieve()
.bodyToMono(String.class)
.retryWhen(Retry.backoff(maxRetries, minBackoff).filter(ctx -> {
return ctx.exception() instanceof RestClientException && ctx.exception().statusCode == 500;
}))
It's worth mentioning that retryWhen wraps the source exception into the RetryExhaustedException. If you want to 'restore' the source exception you can use the reactor.core.Exceptions util:
.onErrorResume(throwable -> {
if (Exceptions.isRetryExhausted(throwable)) {
throwable = throwable.getCause();
}
return Mono.error(throwable);
})
I'm not sure, what spring version you are using, in 2.1.4 I have this:
client.post()
.syncBody("test")
.retrieve()
.bodyToMono(String.class)
.retryBackoff(numretries, firstBackoff, maxBackoff, jitterFactor);
... so that's exactly what you want, right?
I'm currently trying it with Kotlin Coroutines + Spring WebFlux:
It seems the following is not working:
suspend fun ClientResponse.asResponse(): ServerResponse =
status(statusCode())
.headers { headerConsumer -> headerConsumer.addAll(headers().asHttpHeaders()) }
.body(bodyToMono(DataBuffer::class.java), DataBuffer::class.java)
.retryWhen {
Retry.onlyIf { ctx: RetryContext<Throwable> -> (ctx.exception() as? WebClientResponseException)?.statusCode in retryableErrorCodes }
.exponentialBackoff(ofSeconds(1), ofSeconds(5))
.retryMax(3)
.doOnRetry { log.error("Retry for {}", it.exception()) }
)
.awaitSingle()
AtomicInteger errorCount = new AtomicInteger();
Flux<String> flux =
Flux.<String>error(new IllegalStateException("boom"))
.doOnError(e -> {
errorCount.incrementAndGet();
System.out.println(e + " at " + LocalTime.now());
})
.retryWhen(Retry
.backoff(3, Duration.ofMillis(100)).jitter(0d)
.doAfterRetry(rs -> System.out.println("retried at " + LocalTime.now() + ", attempt " + rs.totalRetries()))
.onRetryExhaustedThrow((spec, rs) -> rs.failure())
);
We will log the time of errors emitted by the source and count them.
We configure an exponential backoff retry with at most 3 attempts and no jitter.
We also log the time at which the retry happens, and the retry attempt number (starting from 0).
By default, an Exceptions.retryExhausted exception would be thrown, with the last failure() as a cause. Here we customize that to directly emit the cause as onError.
I am new to spring 5.
1) How I can log the method params which are Mono and flux type without blocking them?
2) How to map Models at API layer to Business object at service layer using Map-struct?
Edit 1:
I have this imperative code which I am trying to convert into a reactive code. It has compilation issue at the moment due to introduction of Mono in the argument.
public Mono<UserContactsBO> getUserContacts(Mono<LoginBO> loginBOMono)
{
LOGGER.info("Get contact info for login: {}, and client: {}", loginId, clientId);
if (StringUtils.isAllEmpty(loginId, clientId)) {
LOGGER.error(ErrorCodes.LOGIN_ID_CLIENT_ID_NULL.getDescription());
throw new ServiceValidationException(
ErrorCodes.LOGIN_ID_CLIENT_ID_NULL.getErrorCode(),
ErrorCodes.LOGIN_ID_CLIENT_ID_NULL.getDescription());
}
if (!loginId.equals(clientId)) {
if (authorizationFeignClient.validateManagerClientAccess(new LoginDTO(loginId, clientId))) {
loginId = clientId;
} else {
LOGGER.error(ErrorCodes.LOGIN_ID_VALIDATION_ERROR.getDescription());
throw new AuthorizationException(
ErrorCodes.LOGIN_ID_VALIDATION_ERROR.getErrorCode(),
ErrorCodes.LOGIN_ID_VALIDATION_ERROR.getDescription());
}
}
UserContactDetailEntity userContactDetail = userContactRepository.findByLoginId(loginId);
LOGGER.debug("contact info returned from DB{}", userContactDetail);
//mapstruct to map entity to BO
return contactMapper.userEntityToUserContactBo(userContactDetail);
}
You can try like this.
If you want to add logs you may use .map and add logs there. if filters are not passed it will return empty you can get it with swichifempty
loginBOMono.filter(loginBO -> !StringUtils.isAllEmpty(loginId, clientId))
.filter(loginBOMono1 -> loginBOMono.loginId.equals(clientId))
.filter(loginBOMono1 -> authorizationFeignClient.validateManagerClientAccess(new LoginDTO(loginId, clientId)))
.map(loginBOMono1 -> {
loginBOMono1.loginId = clientId;
return loginBOMono1;
})
.flatMap(o -> {
return userContactRepository.findByLoginId(o.loginId);
})
I have an okhttp3 (3.9.1) WebSocket instance and would like to view all it's network requests and responses. I tried to add some okhttp3.Interceptor instances to OkHttpClient instance before creating WebSocket on it but had no luck in viewing network activity. Here's sample code which demonstrates what I've tried to do:
package sample
import okhttp3.*
import java.io.IOException
import java.lang.Thread.sleep
fun main(args: Array<String>) {
val listener = object : WebSocketListener() {
override fun onMessage(webSocket: WebSocket?, text: String?) {
println("Got server message: $text")
}
}
val dummyInterceptor = Interceptor { chain ->
val request = chain.request()
val response = chain.proceed(request)
println("Dummy interceptor fired!\n\nRequest: ${request.headers()}\nResponse: ${response.headers()}")
return#Interceptor response
}
val dummyNetworkInterceptor = Interceptor { chain ->
val request = chain.request()
val response = chain.proceed(request)
println("Dummy network interceptor fired!\n\nRequest: ${request.headers()}\nResponse: ${response.headers()}")
return#Interceptor response
}
val okHttpClient = OkHttpClient.Builder()
.addInterceptor(dummyInterceptor)
.addNetworkInterceptor(dummyNetworkInterceptor)
.build()
val request = Request.Builder().url("ws://echo.websocket.org").build()
val webSocket = okHttpClient.newWebSocket(request, listener)
webSocket.send("Hello1!")
webSocket.send("Hello2!")
webSocket.send("Hello3!")
sleep(2000) //Just for this sample to ensure all WS requests done
println("\n\n\tSome network activity\n\n")
okHttpClient.newCall(Request.Builder().get().url("http://echo.websocket.org").build()).enqueue(object : Callback {
override fun onFailure(call: Call?, exc: IOException?) {
println("OnFailure: ${exc?.message}")
}
override fun onResponse(call: Call?, response: Response?) {
println("OnResponse: ${response?.headers()}")
}
})
}
I tried to dive into okhttp3 source code and didn't find any reason why any of my interceptors doesn't fire on WS requests but works perfectly for any OkHttpClient request.
Is it a bug in okhttp3 or am I doing something wrong or it's just not possible to monitor WS requests using okhttp3.Interceptor?
WebSocket calls made with OkHttp don't use the interceptor chains that HTTP calls do, therefore you can't monitor them through interceptors.
I've faced this issue before myself, and so I looked at the source code and found the following then:
The regular HTTP calls go through the getResponseWithInterceptorChain() method in the RealCall class, which quite clearly starts the chained call of interceptors for each request.
The okhttp3.internal.ws package that includes the implementation of the WebSocket handling contains no code related to interceptors.
And really, interceptors catching WebSocket requests wouldn't really make sense in the first place. The Request that you can obtain in an interceptor represents an HTTP request, which WebSocket messages are not.
It isn't possible at this point, there's a feature request open for OkHttp but it isn't getting much traction: https://github.com/square/okhttp/issues/4192