The durchfuehrungen are in a few cases null even if the records are correct and the cogrouping with them works elsewhere.
In the history of the resulting affected ProjektAggregat records no single event contains the depending durchfuehrungen, they’re always null and couldn’t figure it out why.
Function { projekte: KStream<String, ProjektEvent> ->
Function { projektstatus: KStream<String, ProjektStatusEvent> ->
Function { befunde: KStream<String, ProjektBefundAggregat> ->
Function { aufgaben: KStream<String, ProjektAufgabeAggregat> ->
Function { wirtschaftseinheiten: KTable<String, WirtschaftseinheitAggregat> ->
Function { durchfuehrungen: KStream<String, ProjektDurchfuehrungAggregat> ->
Function { gruppen: KStream<String, ProjektGruppeAggregat> ->
Function { mietobjekte: KTable<String, MietobjektAggregat> ->
projekte
.leftJoin(wirtschaftseinheiten)
.leftJoin(mietobjekte)
.cogroup { _, base, current: ProjektAggregat ->
current.copy(
projekt = base.projekt,
wirtschaftseinheit = base.wirtschaftseinheit,
mietobjekt = base.mietobjekt,
projektErstelltAm = base.projektErstelltAm
)
}
.cogroup(projektstatus.groupByKey()) { _, projektstatusEvent, aggregat -> aggregat + projektstatusEvent }
.cogroup(befunde.groupByKey()) { _, befundAggregat, aggregat -> aggregat + befundAggregat }
.cogroup(aufgaben.groupByKey()) { _, aufgabeAggregat, aggregat -> aggregat + aufgabeAggregat }
.cogroup(durchfuehrungen.groupByKey()) { _, durchfuehrungAggregat, aggregat -> aggregat + durchfuehrungAggregat }
.cogroup(gruppen.groupByKey()) { _, gruppeAggregat, aggregat -> aggregat + gruppeAggregat }
.aggregate({ ProjektAggregat() }, Materialized.`as`(projektStoreSupplier))
.toStream()
.filterNot { _, projektAggregat -> projektAggregat.projekt == null }
.transform({ EventTypeHeaderTransformer() })
}
}
}
}
}
}
}
}
Event if I change the stream to use leftJoins instead of cogrouops there are missing durchfuehrungen:
Function { projektstatus: KStream<String, ProjektStatusEvent> ->
Function { befunde: KStream<String, ProjektBefundAggregat> ->
Function { aufgaben: KStream<String, ProjektAufgabeAggregat> ->
Function { wirtschaftseinheiten: GlobalKTable<String, WirtschaftseinheitAggregat> ->
Function { durchfuehrungen: KStream<String, ProjektDurchfuehrungAggregat> ->
Function { gruppen: KStream<String, ProjektGruppeAggregat> ->
Function { mietobjekte: GlobalKTable<String, MietobjektAggregat> ->
projekte
.filterNot { _, projektEvent -> projektEvent.action == CREATE_REQUEST }
.leftJoin(wirtschaftseinheiten)
.leftJoin(mietobjekte)
.leftJoin(projektstatus.toTable()) { aggregat, projektstatusEvent ->
projektstatusEvent?.let { aggregat + projektstatusEvent } ?: aggregat
}
.leftJoin(befunde.toTable()) { aggregat, befundAggregat -> befundAggregat?.let { aggregat + befundAggregat } ?: aggregat }
.leftJoin(aufgaben.toTable()) { aggregat, aufgabeAggregat -> aufgabeAggregat?.let { aggregat + aufgabeAggregat } ?: aggregat }
.leftJoin(durchfuehrungen.toTable()) { aggregat, durchfuehrungAggregat ->
durchfuehrungAggregat?.let { aggregat + durchfuehrungAggregat } ?: aggregat
}
.leftJoin(
gruppen.toTable(),
{ aggregat, gruppeAggregat -> gruppeAggregat?.let { aggregat + gruppeAggregat } ?: aggregat },
Materialized.`as`(ProjektStore.NAME)
)
.toStream()
.filterNot { _, projektAggregat -> projektAggregat.projekt == null }
.process(ProcessorSupplier { EventTypeHeaderProcessor() })
}
}
}
}
}
}
}
}
The cause is a bug in the event type filtering mechanism of Spring Cloud Stream which causes several matching records to not to arrive at the stream. After deactivating the feature all desired records arrive.
https://github.com/spring-cloud/spring-cloud-stream/issues/2627
Related
That's my code:
Flux.fromStream(ids.stream())
.flatMap(id -> Mono.defer(() -> {
String entity = myRepository.findById(id);
return Mono.just(entity);
})
)
.collectList()
I'd like to know how to properly catch each exception could be raised into myRepository.findById.
What about:
Scheduler scheduler = Schedulers.fromExecutor(...);
Flux.fromStream(ids.stream())
.flatMap(id -> Mono.defer(() -> {
String entity = myRepository.findById(id);
return Mono.just(entity);
}).subscribeOn(scheduler),
10
)
.collectList()
.block();
Flux.range(1,100).delayElements(Duration.ofSeconds(10)).subscribe(i-> System.out.println(i));
When the Flux emmiter to 10 , I want to delayElement to 1 minute to this subscribe
The background was when I read some data from mongoDB and write to Elasticsearch,
but I want to dynamic to control the reading speed and don't want to exhaust the mongodb resource.
Flux<List<Document>> readFromMongoDB = getFromMongoDB();
product_2014.subscribe(new BaseSubscriber<List<Document>>() {
int counter;
#Override
protected void hookOnSubscribe(Subscription subscription) {
subscription.request(1);
}
#SneakyThrows
#Override
protected void hookOnNext(List<Document> value) {
Thread.sleep(1000);
if (counter == 1) {
counter = 0;
}
else {
counter++;
}
upstream().request(0);
upstream().cancel();
log.info("aaaaaaaaaaa");
}
});
ParallelFlux<List<Document>> getFromMongoDB(String product,
int size,
MongoDatabase mongoDatabase,
int parallel,
Duration duration) {
Publisher<Long> publisher = mongoDatabase.getCollection(product)
.countDocuments();
Mono<Long> count = Mono.from(publisher);
return count.flatMapMany(l -> {
log.info("split counter");
return Flux.range(0, (int) (l / size) + 1);
})
.log().doOnSubscribe(subscription -> {
log.info("doOnSubscribe");
})
.parallel(parallel,1)
.runOn(scheduler)
.doOnNext(integer -> log.info("get page = {}", integer))
.concatMap(page -> {
log.info("page in {}", page);
FindPublisher<Document> limit =
mongoDatabase.getCollection(product)
.find(Document.class)
.skip(page * size)
.limit(size);
Mono<List<Document>> listDocument = Flux.from(limit)
.publishOn(scheduler)
.collectList()
.doOnNext(list -> {
log.info("{} in list",
page);
});
return listDocument;
});
}
Why I can cacel the subscrition.And the Flux still emit the page element?
And how should I do?
I want to paralle to read from mongodb and dynamic control the reading speed.
I only see it like this:
Flux.range(1, 100)
.flatMap(el -> {
Mono<Integer> elMono = Mono.just(el);
if (el <= 10) {
return elMono.delayElement(Duration.ofMinutes(1));
}
else {
return elMono.delayElement(Duration.ofSeconds(10));
}
})
.subscribe(i-> System.out.println(i));
You examine each element and decide how to delay exactly this one.
I'm trying to add a conditional Retry for WebClient with Kotlin Coroutines + WebFlux.fn + reactor-addons:
suspend fun ClientResponse.asResponse(): ServerResponse =
status(statusCode())
.headers { headerConsumer -> headerConsumer.addAll(headers().asHttpHeaders()) }
.body(bodyToMono(DataBuffer::class.java), DataBuffer::class.java)
.retryWhen {
Retry.onlyIf { ctx: RetryContext<Throwable> -> (ctx.exception() as? WebClientResponseException)?.statusCode in retryableErrorCodes }
.exponentialBackoff(ofSeconds(1), ofSeconds(5))
.retryMax(3)
.doOnRetry { log.error("Retry for {}", it.exception()) }
)
.awaitSingle()
also adding a condition before the retry
if (statusCode().isError) {
body(
BodyInserters.fromPublisher(
Mono.error(StatusCodeError(status = statusCode())),
StatusCodeException::class.java
)
)
} else {
body(bodyToMono(DataBuffer::class.java), DataBuffer::class.java)
}
Call looks like:
suspend fun request(): ServerResponse =
webClient/*...*/
.awaitExchange()
.asResponse()
This spring webclient: retry with backoff on specific error gave me the hint to answer the question:
.awaitExchange() returns the ClientResponse and not Mono<ClientReponse>
This means my retry was acting on bodyToMono instead of the operation of exchange().
The solution now looks like
suspend fun Mono<ClientResponse>.asResponse(): ServerResponse =
flatMap {
if (it.statusCode().isError) {
Mono.error(StatusCodeException(status = it.statusCode()))
} else {
it.asResponse()
}
}.retryWhen(
Retry.onlyIf { ctx: RetryContext<Throwable> ->
(ctx.exception() as? StatusCodeException)?.shouldRetry() ?: false
}
.exponentialBackoff(ofSeconds(1), ofSeconds(5))
.retryMax(3)
.doOnRetry { log.error { it.exception() } }
).awaitSingle()
private fun ClientResponse.asResponse(): Mono<ServerResponse> =
status(statusCode())
.headers { headerConsumer -> headerConsumer.addAll(headers().asHttpHeaders()) }
.body(bodyToMono(DataBuffer::class.java), DataBuffer::class.java)
I'm experimenting with Spring-Boot 2 Kotlin Reactive right now. One of my endpoints should be able to resolve a phone number from a MongoDB to a person, company or department.
As a result, the display name should e.g. for a phone, be returned as a string.
The result can be changed with parameters in the request:
http://localhost:8080/api/v1/numbers/find/xxxxxxxxxxx?asString=true&firstname=2&lastname=1&company=3&department=4
The values of firstname, lastname, company, and department specify the order in the display name.
If "xxxxxxxxxxx" is the number of a person, the response should look like this:
"lastname, firstname, company, department"
I've found a solution to that, but I wonder if there is not a better way.
fun findNumberOwner(serverRequest: ServerRequest) =
numberRepository.findByNumber(serverRequest.pathVariable("number")).flatMap { n ->
serverRequest.queryParam("asString")
.filter {
it.trim()
.isNotEmpty().and(it.toBoolean())
}
.map { searchNumberWithStringResult(n, serverRequest.queryParam("firstname"), serverRequest.queryParam("lastname"), serverRequest.queryParam("company"), serverRequest.queryParam("department")) }
.orElse(searchNumberWithJsonResult(n))
}.switchIfEmpty(ServerResponse.notFound().build())
private fun searchNumberWithStringResult(number: Number, firstname: Optional<String>, lastname: Optional<String>, company: Optional<String>, department: Optional<String>): Mono<ServerResponse> {
val ownerId = number.numberOwner
if (ownerId.startsWith("CO-")) {
return ok().body(companyRepository.findById(ownerId).map { it.name })
} else if (ownerId.startsWith("DE-")) {
return ok().body(departmentRepository.findById(ownerId)
.map { dep ->
companyRepository.findById(dep.company)
.map {
it.name
}.zipWith(dep.toMono())
}.flatMap { it ->
it
}.map {
it.t1 + ", " + it.t2.name
})
} else {
val templateMap = mapOf(
Pair("firstname", firstname.orElse("2").toInt()),
Pair("lastname", lastname.orElse("1").toInt()),
Pair("company", company.orElse("0").toInt()),
Pair("department", department.orElse("0").toInt())
)
return ok().body(personRepository.findById(ownerId)
.map { person ->
companyRepository.findById(person.company)
.map {
it.name
}
.switchIfEmpty(Mono.just(""))
.zipWith(person.toMono())
}.flatMap { it ->
it
}.map { t ->
departmentRepository.findById(t.t2.department).map {
mapOf(
Pair(templateMap["firstname"], t.t2.firstName) as Pair<Int, String>,
Pair(templateMap["lastname"], t.t2.lastName) as Pair<Int, String>,
Pair(templateMap["company"], t.t1) as Pair<Int, String>,
Pair(templateMap["department"], it.name) as Pair<Int, String>)
.filter { it.key > 0 }.filter { !it.value.equals("") }.toSortedMap()
}
.switchIfEmpty(Mono.just(
mapOf(
Pair(templateMap["firstname"], t.t2.firstName) as Pair<Int, String>,
Pair(templateMap["lastname"], t.t2.lastName) as Pair<Int, String>,
Pair(templateMap["company"], t.t1) as Pair<Int, String>)
.filter { it.key > 0 }.filter { !it.value.equals("") }.toSortedMap()
))
}.flatMap { it -> it }
.map { it }
.map { myMap ->
var finalDisplayString = ""
myMap.forEach {
finalDisplayString += buildDisplayString(it.key, myMap.size, it.value)
}
finalDisplayString
})
}
}
private fun buildDisplayString(index: Int, maxSize: Int, value: String?): String {
if (value == null) return ""
if (index == 0) {
return "$value, "
} else if (index == maxSize) {
return "$value"
} else {
return "$value, "
}
}
I m developing a media player app and I m using VideoService for this purpose, my issue is the app life cycle when its minmized in backgroud while playing first its pause the video and i handle this in pause handler, when i maximize the app i call videoservice.play() but it play for a second and the whole app stuck in black screen and crash at the end.
if i dont call videoservice.play it will just crash, here my sample code.
private void initVideoService(VideoService service) {
videoService = service;
videoService.setPosition(Pos.CENTER, 0, 40, 0, 40);
videoService.setControlsVisible(true);
videoService.fullScreenProperty().addListener((obs, ov, nv) -> MobileApplication.getInstance().getAppBar().setVisible(!nv)); MobileApplication.getInstance().getAppBar().getActionItems().addAll(playButton, stopButton, fullScreenButton);
videoService.statusProperty().addListener((obs, ov, nv) -> {
System.out.println(String.format("FullScreenCondition : %b", nv != MediaPlayer.Status.PLAYING && nv != MediaPlayer.Status.PAUSED));
fullScreenButton.setDisable(nv != MediaPlayer.Status.PLAYING && nv != MediaPlayer.Status.PAUSED);
if (videoService.statusProperty().get() == MediaPlayer.Status.PLAYING) {
System.out.println("Video is Playing");
playButton.setGraphic(MaterialDesignIcon.PAUSE.graphic());
videoService.setFullScreen(true);
} else if (videoService.statusProperty().get() == MediaPlayer.Status.PAUSED) {
System.out.println("Video is paused");
playButton.setGraphic(MaterialDesignIcon.PLAY_ARROW.graphic());
videoService.setFullScreen(false);
} else {
System.out.println("Video is Stopped");
playButton.setGraphic(MaterialDesignIcon.PLAY_ARROW.graphic());
videoService.setFullScreen(false);
}
});
Services.get(LifecycleService.class).ifPresent(s -> {
s.addListener(LifecycleEvent.PAUSE, () -> {
System.out.println("paused");
videoService.pause();
});
s.addListener(LifecycleEvent.RESUME, () -> {
System.out.println("Resumed");
videoService.play();
});
});
}
public void navToEpisode(Episode epi) {
Services.get(VideoService.class).ifPresent(service -> {
initVideoService(service);
});
RestClient.execute(new RestOperation() {
#Override
public void run() {
String videoDownloadLink = RestUtils.getVideoDownloadLink(epi.getLinkToEpisode());
try {
publish(() ->
{
if (Platform.isAndroid() || Platform.isIOS()) {
videoService.getPlaylist().add(videoDownloadLink);
videoService.play();
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
});
}