I am writing tests for the method provide below.
`
class ScrapedRecipeCache #Autowired constructor(private val cache: RecipeScrapingCacheService,
private val recipeService: RecipeService) : ScrapedRecipeProvider {
override fun provide(request: ScrapingRequest): Flux<ScrapedRecipe> =
cache.retrieve(request.link)
.doOnNext { println(it) }
.flatMap { (link, _, recipeHash, error) ->
recipeService.findByHash(recipeHash)
.map { ScrapedRecipe(it, link, error)}
.switchIfEmpty(cache.remove(request.link).then(Mono.empty()))
}
.flux()
}
`
The test looks as follows:
private val recipeFetched = Recipe("Tortellini", RecipeDifficulty.EASY, 15.0)
val cacheContents = RecipeScrapingResource("www.google.com", ScrapingOrigin.JAMIE_OLIVER, recipeFetched.hash,
mutableListOf(
pl.goolash.core.Exception("aa", ErrorType.WARNING, LocalDateTime.MIN)
))
val request = ScrapingRequest("www.google.com", ScrapingOrigin.JAMIE_OLIVER, 4)
#BeforeEach
fun setUp() {
given(cache.retrieve("www.google.com")).willReturn(Mono.just(cacheContents))
given(recipeService.findByHash(recipeFetched.hash)).willReturn(Mono.just(recipeFetched))
}
#Test
#DisplayName("Then return data fetched from service and don't delete cache")
fun test() {
cacheFacade.provide(request)
.test()
.expectNext(ScrapedRecipe(recipeFetched, "www.google.com", cacheContents.error!!))
.expectComplete()
.verify()
BDDMockito.verify(cache, BDDMockito.never()).remove(request.link)
}
The test fails because cache.remove(request.link) is called. To my understanding (or from what I managed to gather from documentation) switchIfEmpty, should only be fired when recipeService.findByHash returns Mono.empty(). However the debugger shows that it returns mocked value of Mono.just(fetchedRecipe).
The interesting thing is that when I replace
.switchIfEmpty(cache.remove(request.link).then(Mono.empty()))
with
.switchIfEmpty(Mono.just(1).doOnNext{println("weeee")}.then(Mono.empty()))
Then weee is not printed hence it behaves as expected, that is switchIfEmpty is not fired.
Furthermore the tested issue runs properly in integration test and does not clear the cache.
Reactor version : 3.1.0-RC1
Other notable details: Spring Boot 2.0.0-M4, Mockito-core:2.10, junit 5, project is written in kotlin
The question is, does anybody see anything wrong with this? Because I have spent two days over and still have no clue why this behaves so bizzarely.
Finally I found out how to make this work.
In order to remedy it:
override fun provide(request: ScrapingRequest): Flux<ScrapedRecipe> =
cache.retrieve(request.link)
.flatMap { (link, _, recipeHash, error) ->
recipeService.findByHash(recipeHash)
.map { ScrapedRecipe(it, link, error) }
.switchIfEmpty(Mono.just(1)
.flatMap { cache.remove(request.link) }
.then(Mono.empty()))
}
.flux()
You can see how using flatMap to execute the asynch work does the job, even if this is not the neatest implementation, it revealed to me quite an interesting mechanism hidden here.
Related
You might wonder why don't I use #Transaction which spring already provided. I wanted too, but I'm doing event-sourcing anything that happened can't be changed or rollback. So I need to manually publish another compensation event which solve the previous one that already happened. More over I'm using project reactor so I'm kinda stuck on how should I design to make it works.
Let's say in case of hotel reservation I would have these process. So when the makePayment is failed I should rollback transaction by publish compensation events like this.
// pb stands for publish
Mono.just("start_transaction")
.flatMap { reserveHotelRoom(roomId) } pb RoomReservedEvent v ^ pb RoomUnreservedEvent
.flatMap { applyDiscount(coupon) } pb CouponRedeemedEvent v ^ pb CouponUnredeemedEvent
.flatMap { makePayment(bankAccount) } pb paymentHasbeenMadeEvent v -> oops error happens xD ^
This is what I have tried so far, but it doesn't work. once error has been thrown it still execute the next process so I have to continue working on it, but hope you get an idea of what I'm trying to do so.
class ReactiveTransaction() {
private var aggregators = Mono.just("_start_transaction_")
private var compensationAction = Mono.just("_start_rollback_")
fun addExecution(execution: Mono<String>, onError: Mono<String>): ReactiveTransaction {
// i don't know how to clone mono object so this is the current solution
val temp = compensationAction.flatMap { onError }
compensationAction = temp
aggregators = aggregators.flatMap { execution }
.onErrorResume { temp }
.checkpoint()
return this;
}
fun execute(): Mono<String> {
return aggregators
}
}
fun main() {
ReactiveTransaction().addExecution(reserveHotel, unreserveHotel)
.addExecution(applyDiscount, unapplyDiscount)
.addExecution(makePayment, cancelPayment)
.execute()
}
Is this design valid what should I concern ? or any library that can handle this problem recommend?
update: I use mongodb as my event store with reactive repository
I'm trying to achieve functionality: I have a rest endpoint that calls code that execution can take a lot of time. My idea to improve experience for now is to wrap that piece of code as a new thread, wait for completion or for some max time to elapse and return an appropriate message. Wrapped code should be completed even through endpoint already send message back. Current implementation looks like this:
private const val N = 1000
private const val MAX_WAIT_TIME = 5000
#RestController
#RequestMapping("/long")
class SomeController(
val service: SomeService,
) {
private val executor = Executors.newFixedThreadPool(N)
#PostMapping
fun longEndpoint(#RequestParam("someParam") someParam: Long): ResponseEntity<String> {
val submit = executor.submit {
service.longExecution(someParam)
}
val start = System.currentTimeMillis()
while (System.currentTimeMillis() - start < MAX_WAIT_TIME) {
if (submit.isDone)
return ResponseEntity.ok("Done")
}
return ResponseEntity.ok("Check later")
}
}
First question is - waiting on while for time seems wrong, we don't release thread, can it be improved?
More important question - how to rewrite it to Kotlin coroutines?
My attempt, simple without returning as soon as task is done, looked like this:
#PostMapping
fun longEndpoint(#RequestParam("someParam") someParam: Long): ResponseEntity<String> = runBlocking {
val result = async {
withContext(Dispatchers.Default) {
service.longExecution(someParam)
}
}
delay(MAX_WAIT_TIME)
return#runBlocking ResponseEntity.ok(if(result.isCompleted) "Done" else "Check later")
}
But even through correct string is returned, answer is not send until longExecution is done. How to fix that, what am I missing? Maybe coroutines are bad application here?
There are several problems with your current coroutines attempt:
you are launching your async computation within runBlocking's scope, so the overall endpoint method will wait for child coroutines to finish, despite your attempt at return-ing before that.
delay() will always wait for MAX_WAIT_TIME even if the task is done quicker than that
(optional) you don't have to use runBlocking at all if your framework supports async controller methods (Spring WebFlux does support suspend functions in controllers)
For the first problem, remember that every time you launch a coroutine that should outlive your function, you have to use an external scope. coroutineScope or runBlocking are not appropriate in these cases because they will wait for your child coroutines to finish.
You can use the CoroutineScope() factory function to create a scope, but you need to think about the lifetime of your coroutine and when you want it cancelled. If the longExecution function has a bug and hangs forever, you don't want to leak the coroutines that call it and blow up your memory, so you should cancel those coroutines somehow. That's why you should store the scope as a variable in your class and cancel it when appropriate (when you want to give up on those operations).
For the second problem, using withTimeout is very common, but it doesn't fit your use case because you want the task to keep going even after you timeout waiting for it. One possible solution would be using select clauses to either wait until the job is done, or wait for some specified maximum time:
// TODO call scope.cancel() somewhere appropriate (when this component is not needed anymore)
val scope = CoroutineScope(Job())
#PostMapping
fun longEndpoint(#RequestParam("someParam") someParam: Long): ResponseEntity<String> {
val job = scope.launch {
longExecution()
}
val resultText = runBlocking {
select {
job.onJoin() { "Done" }
onTimeout(MAX_WAIT_TIME) { "Check later" }
}
}
return ResponseEntity.ok(resultText)
}
Note: I'm using launch instead of async because you don't seem to need the return value of longExecution here.
If you want to solve the problem #3 too, you can simply declare your handler suspend and remove runBlocking around the select:
// TODO call scope.cancel() somewhere appropriate (when this component is not needed anymore)
val scope = CoroutineScope(Job())
#PostMapping
suspend fun longEndpoint(#RequestParam("someParam") someParam: Long): ResponseEntity<String> {
val job = scope.launch {
longExecution()
}
val resultText = select {
job.onJoin() { "Done" }
onTimeout(MAX_WAIT_TIME) { "Check later" }
}
return ResponseEntity.ok(resultText)
}
Note that this requires spring-boot-starter-webflux instead of spring-boot-starter-web.
Your implementation always waits for MAX_WAIT_TIME. This might work:
#PostMapping
fun longEndpoint(#RequestParam("someParam") someParam: Long): ResponseEntity<String> = runBlocking {
try {
withTimeout(MAX_WAIT_TIME) {
async {
withContext(Dispatchers.Default) {
service.longExecution(someParam)
}
}
}
} catch (ex: CancellationException) {
return#runBlocking ResponseEntity.ok("Check later")
}
return#runBlocking ResponseEntity.ok("Done")
}
Although I'm not sure if there will be any unwanted side effects because it seems that this will cancel the coroutine when it reaches MAX_WAIT_TIME. Read more about it here:
Cancellation and timeouts
I have an endpoint exposed, that is launching a coroutine:
val apiCall = ApiCall()
#GetMapping("/example")
fun example(#RequestParam paramExample:String):Int{
GlobalScope.launch{
return apiCall.callApi(paramExample)
}
}
This function is calling another external API, using Retrofit:
suspend fun callApi(param:String):Int{
var tot_records =0
val retrofit: Retrofit = Retrofit.Builder()
.baseUrl(appProperties.sampleUrl)
.addConverterFactory(GsonConverterFactory.create())
.build()
val service = retrofit.create<ResponseService>(ResponseService::class.java)
service.getResponse().enqueue(object : Callback<Response> {
override fun onFailure(call: Call<Response>, throwable: Throwable) {
println("Error")
println(throwable.stackTrace)
}
override fun onResponse(call: Call<Response>, response: Response<Response>) {
println("OK")
println(response.body())
println("Tot records")
tot_records = response.body()?.tot_records!!
}
})
return tot_records
}
The problem is that I can't launch this, the error is: 'return' is not allowed here
Any idea how to fix it and whats is happening?
Thanks for your help
It seems like you can't decide if you want your code to be synchronous (so code waits for its subtasks to finish before continuing) or asynchronous (it launches operations in the background). You intend to return a result from example(), so you need it to be synchronous, but you immediately use launch() to invoke callApi() asynchronously. The same in callApi() - you intend to return from it (so synchronous), but you invoke Retrofit using callbacks (so asynchronous). Note that callApi() has exactly the same problem as example(). Even if it compiles, it still does not really work properly. It always returns 0, because tot_records is returned before being set.
You have to decide between asynchronous and synchronous and stick to it. If you want to go fully asynchronous, then you need to redesign both callApi() and example() to return their results either with callbacks or futures.
However, I suggest going fully synchronous, utilizing Kotlin suspend functions. Make all functions suspend: example(), callApi() (it is already) and ResponseService.getResponse(). The last one will look something like:
suspend fun getResponse(): Response
Then remove GlobalScope.launch(), and almost everything inside enqueue(). Instead, service.getResponse() will return Response object directly, so you can just return its tot_records property.
Also note that in your original code you ignored failures. After above change service.getResponse() will throw exceptions on failures, so you have to handle them.
This solution seems that works:
This is the endpoint declaration:
#GetMapping("/example")
suspend fun example(#RequestParam param:String):CustomResponse{
return coroutineScope {
val job = async{apiCall.callApi(param)}
job.await()
}
}
And this is my function that is calling an external API:
suspend fun callApi(param:String):CustomResponse{
var responseCustom = CustomResponse()
val retrofit: Retrofit = Retrofit.Builder()
.baseUrl(appProperties.reservationUrl)
.addConverterFactory(GsonConverterFactory.create())
.build()
val service = retrofit.create<CustomResponseService>(CustomResponseService::class.java)
responseCustom = service.getResponse(appProperties.token, param).execute().body()!!
return responseCustom
}
I am trying to convert a REST service from the Spring 5 Reactive style to an async Kotlin Coroutine style.
I followed several different guides/tutorials on how this should work but I must be doing something wrong.
I get a compile error trying to turn a single object into a Flow, whereas the guides I'm following dont seem to do this at all.
Any pointers or otherwise very appreciated!
Router:
#Bean
fun mainRouter(handler: EobHandler) = coRouter {
GET("/search", handler::search)
GET("/get", handler::get)
}
Handler:
suspend fun search(request: ServerRequest): ServerResponse {
val eobList = service.search()
return ServerResponse.ok().bodyAndAwait(eobList)
}
suspend fun get(request: ServerRequest): ServerResponse
val eob = service.get()
return ServerResponse.ok().bodyAndAwait(eob); // compile error about bodyAndAwait expecting a Flow<T>
}
Service:
override fun search(): Flow<EOB> {
return listOf(EOB()).asFlow()
}
//
override suspend fun get(): EOB? {
return EOB()
}
If curious, here are some of the guides I've based my code on:
https://www.baeldung.com/spring-boot-kotlin-coroutines
https://docs.spring.io/spring/docs/5.2.0.M1/spring-framework-reference/languages.html#how-reactive-translates-to-coroutines
https://medium.com/#hantsy/using-kotlin-coroutines-with-spring-d2784a300bda
I was able to get this to compile by changing
return ServerResponse.ok().bodyAndAwait(eob);
to
return eob?.let { ServerResponse.ok().bodyValueAndAwait(it) } ?: ServerResponse.notFound().buildAndAwait()
guess it's something to do with type-safety of Kotlin - I was not returning a nullable object I think
I am trying to make requests using WebClient in parallel, but I have no clue how to go about that,
because no matter what I do, the code is not waiting for requests to finish. If I execute just one request though (Commented fragment), everything works fine. Can someone help me with that?
#RequestMapping(method = [RequestMethod.POST], path = ["/upload/{batchId}"])
fun uploadFile(#RequestPart("file") file: Mono<FilePart>,
#PathVariable("batchId") batchId:String,
#RequestHeader("FILE-SIZE") fileSize:Int): Mono<ServiceResponse> {
val webClient = WebClient.create(commandEndpoint)
// return webClient.put().uri(seriesPath).retrieve().bodyToMono(String::class.java).map { ServiceResponse(it,0) }
return file.map{it.transferTo(Paths.get(storagePath,"excel"))}
.map{excelWorkbookToMetadata(WorkbookFactory.create(Paths.get(storagePath,"excel").toFile()))}
.flatMapMany{Flux.fromIterable(it)}
.flatMap {
it.transactionId = batchId
when (it) {
is SeriesMetadata -> webClient.put().uri(seriesPath,it.id)
.body(BodyInserters.fromObject(it))
.retrieve()
.onStatus({ it == HttpStatus.BAD_REQUEST },{
println("ERROR")
Mono.error(RuntimeException("blah")) }).toMono()
else -> Mono.error(NotImplementedError(""))
}
}
.collectList()
.map {ServiceResponse(batchId, it.size*2) }
}
So it seems, that collectList() filters out empty mono that are returned in case the body of the response is empty. The solution is basically, either to use Mono.defaultIfEmpty() method, or change retrieve() to exchange() which always returns something. At least that's what helped me.