I want to use the spring cloud stream api to aggreate events from a topic.
Therefore i use as input a KStream.
KStream<Object, LoggerCreatedMessage>
Now i want to use an aggregator to store my new Object in a KeyValue Store, so i use following code:
input
.map((key, value) -> {
return new KeyValue<>(value.logger_id,value);
})
/*.groupBy(
(s, loggerEvent) -> loggerEvent.logger_id,
Serialized.with(null, loggerEventSerde))*/
.groupByKey()
.aggregate(
String::new,
(s, loggerEvent, vr) -> {
return vr;
},
Materialized.<String, String, KeyValueStore<Bytes, byte[]>>as(STORE_NAME).withKeySerde(Serdes.String()).
withValueSerde(Serdes.String())
);
Why can i only use a String as an Initializer is it not possible to use any Object?
Instead of String::new i wanted to use LoggerDomain::new, but i only get this error message:
Bad return type in method reference: cannot convert LoggerDomain to VR
Do i miss something?
You define <key,value> as <String, String> via Materialized.<String, String, KeyValueStore<Bytes, byte[]>> -- if you value type should be LoggerDomain, it should be Materialized.<KeyType, LoggerDomain, KeyValueStore<Bytes, byte[]>>().
Note that you need to provide a custom Serde for LoggerDomain for this case to Materialized, too.
Related
Because #EnableBinding and #StreamListener are deprecated, I need to migrate existing code to the new model, however I could not find any information on whether the argument mapping available in Spring Cloud Stream is still supported and/or any clean workarounds.
My original method:
#StreamListener("mysource)
public void processMessage(byte[] rawMessage, #Header(required = false, value = KafkaHeaders.RECEIVED_MESSAGE_KEY) byte[] rawKey) {
processMessage(rawMessage, rawKey);
}
I managed to convert this to work as follows:
#Bean(name = "mysource")
public Consumer<Message<?>> mySource() {
return message -> {
byte[] rawMessage = message.getPayload().toString().getBytes();
byte[] rawKey = (byte[]) message.getHeaders().get("kafka_receivedMessageKey");
processMessage(rawMessage, rawKey);
};
}
However, what I would prefer is one that maximizes framework support with respect to argument mapping and/or automatic type conversions.
I attempted:
#Bean(name = "mysource")
public BiConsumer<Message<byte[]>, MessageHeaders> mySource() {
return (message, headers) -> {
byte[] rawMessage = message.getPayload();
byte[] rawKey = (byte[]) headers.get("kafka_receivedMessageKey");
processMessage(rawMessage, rawKey);
};
}
But this gives an error at startup: FunctionConfiguration$FunctionBindingRegistrar.afterPropertiesSet - The function definition 'mysource' is not valid. The referenced function bean or one of its components does not exist
I'm also aware that along with Supplier and Consumer, Function is also available, but I'm not sure how to use a Function in this case instead of a BiConsumer or if it's possible, so looking for examples on how to do this type of migration seamlessly and elegantly with respecting to consuming and producing messages plus headers from/to Kafka.
I am working on spring reactive and need to call multiple calls sequentially to other REST API using webclient. The issue is I am able to call multiple calls to other Rest API but response am not able to read without subscribe or block. I can't use subscribe or block due to non reactive programming. Is there any way, i can merge while reading the response and send it as flux.
Below is the piece of code where I am stuck.
private Flux<SeasonsDto> getSeasonsInfo(List<HuntsSeasonsMapping> l2, String seasonsUrl) {
for (HuntsSeasonsMapping s : l2)
{
List<SeasonsJsonDto> list = huntsSeasonsProcessor.appendSeaosonToJson(s.getSeasonsRef());
for (SeasonsJsonDto sjdto:list)
{
Mono<SeasonsDto> mono =new SeasonsAdapter("http://localhost:8087/").callToSeasonsAPI(sjdto.getSeasonsRef());
//Not able to read stream without subscribe an return as Flux object
}
public Mono<SeasonsDto> callToSeasonsAPI(Long long1) {
LOGGER.debug("Seasons API call");
return this.webClient.get().uri("hunts/seasonsInfo/"
+long1).header("X-GoHunt-LoggedIn-User",
"a4d4b427-c716-458b-9bb5-9917b6aa30ff").retrieve().bodyToMono(SeasonsDto.class);
}
Please help to resolve this.
You need to combine the reactive streams using operators such as map, flatMap and concatMap.
private Flux<SeasonsDto> getSeasonsInfo(List<HuntsSeasonsMapping> l2, String seasonsUrl) {
List<Mono<SeasonsDto>> monos = new ArrayList<>();
for (HuntsSeasonsMapping s : l2) {
List<SeasonsJsonDto> list = huntsSeasonsProcessor.appendSeaosonToJson(s.getSeasonsRef());
for (SeasonsJsonDto sjdto:list) {
Mono<SeasonsDto> mono =new SeasonsAdapter("http://localhost:8087/").callToSeasonsAPI(sjdto.getSeasonsRef());
//Not able to read stream without subscribe an return as Flux object
monos.add(mono);
}
}
return Flux.fromIterable(monos).concatMap(mono -> mono);
}
This can further be improved using the steam API, which I suggest you look into, but I didn't want to change too much of your existing code.
I have figured how to do this. I have completely rewrite the code and change in reactive. It means all the for loop has been removed. Below is the code for the same and may be help for others.
public Flux<SeasonsDto> getAllSeasonDetails(String uuid) {
return hunterRepository.findByUuidAndIsPrimaryAndDeleted(uuid, true, false).next().flatMapMany(h1 -> {
return huntsMappingRepository.findByHunterIdAndDeleted(h1.getId(), false).flatMap(k -> {
return huntsMappingRepository.findByHuntReferrenceIdAndDeleted(k.getHuntReferrenceId(), false)
.flatMap(l2 -> {
return huntsSeasonsProcessor.appendSeaosonToJsonFlux(l2.getSeasonsDtl()).flatMap(fs -> {
return seasonsAdapter.callSeasonsAPI(fs.getSeasonsRef(), h1.getId(), uuid).map(k->{
return k;
});
});
});
});
});
}
I'm trying to retrieve Mono from DB and then filter the Compliance List which is inside the PortCall object based on one condition and finally return a Compliance or Mono
Below is my Mongo DB query
#Query("{vesselCode : ?0, arrivalVoyageCode: ?1}")
Mono<PortCall> findDeadlineTimestamp(String vesselCode, String arrivalVoyageCode);
Below is the usage in ServiceImpl to retrieve Mono
Mono<Compliance> cmp = portCallRepository.findDeadlineTimestamp(arrivalVoyageCode, vesselCode)
.doOnNext(p->p.getCompliance().stream()
.filter(c->c.getId().equalsIgnoreCase(compId))).subscribe();
You should use Reactor's operators instead of Java 8 streams.
The expected way to do that is to actually use the map operator along with the filter:
Mono<Compliance> getCompliance() {
return portCallRepository.findDeadlineTimestamp(arrivalVoyageCode, vesselCode)
.map(e -> e.getCompliance())
.filter(c -> c.getId().equalsIgnoreCase(compId));
}
Then, caller will subscribe:
getCompliance().subscribe()
I have the following code:
public Flux<Offer> getAllFilteredOffers(Map<String, String> searchParams) {
Flux<ProductProperties> productProperties = productPropertiesService.findProductPropertiesBySearchCriteria(searchParams);
Flux<Product> products = productService.findProductsByPropertyId(productProperties);
Flux<Product> productsByAvailability = productService.getAllProductsByAvailability(products, searchParams);
Flux<Offer> offers = offerRepository.findByPropertiesIds(productsByAvailability);
return offers;
This method:
productService.getAllProductsByAvailability(products, searchParams);
looks like:
public Flux<Product> getAllProductsByAvailability(Flux<Product> products,
Map<String, String> searchParams) {
How to pass List<Product> to getAllProductsByAvailability to keep non-blocking operations?
I've read that map is blocking and should be avoided.
Maybe something like that?
Flux
.just(productPropertiesService.findProductPropertiesBySearchCriteria(searchParams))
.flatMap(productProperties -> productService.findProductsByPropertyId(productProperties))
.flatMap(products -> productService.getAllProductsByAvailability(Flux.create(products)?????????, searchParams))
???
I'm not expert in Webflux, currently I'm trying to figure out how to handle problems like: I have Flux but in a second step I need to pull some data from the previous Flex<> object - keeping non-blocking stream.
Than you!
I don't know where you read about map, but if you look at the official documenation Webflux map operator there is nothing about blocking, it just uses synchronous function to each item.
Use this code:
productPropertiesService.findProductPropertiesBySearchCriteria(searchParams)
.flatMap(productProperties -> productService.findProductsByPropertyId(productProperties))
.collectList() (1)
.flatMapMany(products -> productService.getAllProductsByAvailability(Flux.fromIterable(products), searchParams)) (2)
1) collect all elements to List and convert to Mono>
2) create FLux from List and provide it as a parameter, flatMapMany transform Mono to Flux
// given a set of Item objects, group them by the managers of creator and owners
Map<String, List<Item>> managersItems =
itemSet.parallelStream().flatMap(item -> {
// get the list of the creator and owners
List<String> users = new ArrayList();
users.add(item.getCreator());
users.addAll(item.getOwners());
return Stream.of(users.toArray(new String[] {})).map(user -> {
LdapUserInfo ldapUser = LdapUserInfoFactory.create(user);
String manager = ldapUser.getManager();
return new AbstractMap.SimpleImmutableEntry<String, Item(manager, item);
});
}).collect(
Collectors.groupingBy(Map.Entry::getKey, Collectors.mapping(Map.Entry::getValue, Collectors.toList())));
This code compiles fine in Eclipse Mars, but gets the following eror in Eclipse Luna:
Type mismatch: cannot convert from Map<Object,List<Object>> to Map<String,List<WeblabInfo>>
If I do not assign the returned to a Map with Map<String, List<Item>> managersItem = in Eclipse Luna, the error is at Map.Entry::getKey and Map.Entry::getValue statement with message:
The type Map.Entry does not define getKey(Object) that is applicable here".
What did I do wrong?
You didn't do anything wrong. Eclipse compiler has problems with type inference that causes these issues. If Luna compatibility is important, you will have to add explicit types to lambda expressions. Try, for example, Map.Entry::<String,Item>getKey
On another note, it's not necessary to convert a List to array to stream it. You can directly call users.stream(). But even creating the List isn't necessary. You can use Stream.concat(Stream.of(item.getCreator()), item.getOwners().stream()) instead (granted, it's a bit unwieldy).
Finally (and most importantly), avoid using parallelStream with blocking code, such as looking up data in an external system. Parallel streams are designed to handle CPU-bound tasks.
I was able to come up with this solution from Misha's answer. This is working with Eclipse Luna Java compiler
Map<String, List<Item>> managersItems = itemSet
.stream()
.<Map.Entry<String, Item>> flatMap(item -> {
return Stream.concat(Stream.of(item.getCreatorLogin()), item.getOwners().stream()).map(
user -> {
LdapUserInfo ldapUser = LdapUserInfoFactory.create(user);
String manager = ldapUser.getManagerLoginName();
return new AbstractMap.SimpleEntry<String, Item>(manager, info);
});
})
.collect(Collectors.groupingBy(Map.Entry<String, Item>::getKey,
Collectors.mapping(Map.Entry<String, Item>::getValue,
Collectors.toList())));