I am working on the api, which takes ids. For the given id, I want to download related data from s3 and put them in a new object lets call it data
class Data {
private List<S3Object> s3Objects;
//getter-setter
}
public Mono<ResponseEntity<Data>> getData(#RequestParam List<String> tagIds){
Data data = new Data();
Flux<S3Object> s3ObjectFlux = Flux.fromStream(tagIds.stream())
.parallel()
.runOn(Schedulers.boundedElastic())
.flatMap(id -> fetchResources(id))
.flatMap(idS3Object -> Mono.just(s3Object))
.ordered((u1, u2) -> u2.hashCode() - u1.hashCode());
//how do i add it in data object to convert Mono<Data>?
}
You need to collect it into a list and then map it to create a Data object as follows:
public Mono<ResponseEntity<Data>> getData(#RequestParam List<String> tagIds){
Flux<S3Object> s3ObjectFlux = Flux.fromStream(tagIds.stream())
.parallel()
.runOn(Schedulers.boundedElastic())
.flatMap(id -> fetchResources(id))
.flatMap(idS3Object -> Mono.just(s3Object))
.ordered((u1, u2) -> u2.hashCode() - u1.hashCode());
Mono<Data> data = s3ObjectFlux.collectList()
.map(s3Objects -> new Data(s3Objects));
}
Creating a constructor that accepts the S3 objects list is helpful:
class Data {
private List<S3Object> s3Objects;
public Data(List<S3Object> s3Objects) {
this.s3Objects = s3Objects;
}
//getter-setter
}
Related
I have the following object:
public class Book {
private Long id;
private Long bookId;
private String bookName;
private String owner;
}
Represented from following table:
Basically, a book can be owned by multiple owners i.e. Owner "a" owns books 1 and 2.
I have a basic function that will when passed a book object, will give its owner(s) in a List.
private List<String> getBookToOwner(Book book) {
List<String> a = new ArrayList<>();
if (book.getOwner() != null && !book.getOwner().isEmpty()) {
a.addAll(Arrays.asList(book.getOwner().split("/")));
}
return a;
}
I want to use that to apply to each book, retrieve their owners and create the following Map.
Map<String, List<Long>> ownerToBookMap;
Like this:
How do I use streams here?
//books is List<Book>
Map<String, List<Long>> ownerToBookMap = books.stream().map(
// apply the above function to get its owners, flatten it and finally collect it to get the above Map object
// Need some help here..
);
You can get the owner list from the book, then flatten the owners and map as pair of bookId and owner using flatMap. Then grouping by owner using groupingBy and collect the list of bookId of owner.
Map<String, List<Long>> ownerToBookMap =
books.stream()
.flatMap(b -> getBookToOwner(b)
.stream()
.map(o -> new AbstractMap.SimpleEntry<>(o, b.getBookId())))
.collect(Collectors.groupingBy(Map.Entry::getKey,
Collectors.mapping(Map.Entry::getValue, Collectors.toList())));
Flatmap the owners into a single one, create entries with key as an single owner and value as a bookId. Then group the structure by the key (owner). Finally use Collectors::mapping to get the List of bookIds instead of the actual entries:
List<Book> books = ...
Map<String, List<Long>> booksByOwner = books.stream()
.flatMap(book -> Arrays.stream(book.getOwner().split("/"))
.map(owner -> new AbstractMap.SimpleEntry<>(owner, book.getBookId())))
.collect(Collectors.groupingBy(
AbstractMap.SimpleEntry::getKey,
Collectors.mapping(AbstractMap.SimpleEntry::getValue, Collectors.toList())));
I use reduce instead of map.
Map<String, List<Long>> ownerToBookMap = books.stream().reduce(
HashMap::new,
(acc,b) -> {
getBookToOwner(b).stream().forEach( o -> {
if (!acc.containsKey(o))
acc.put(o, new ArrayList<Long>());
acc.get(o).put(b.bookId);
});
return acc;
}
).get();
I have an Kafka streams application in which I read from a topic, do aggregation and materialize in a KTable. I then create a Stream and run some logic on the stream. Now in the stream processing, I want to use some data from the aforementioned KTable. Once I start the stream app, how do I get access to the KTable stream again? I don't want to push the KTable to a new Topic.
KStream<String, MyClass> source = builder.stream("my-topic");
KTable<Windowed<String>, Long> kTable =
source.groupBy((key, value) -> value.getKey(),
Grouped.<String, MyClass >as("repartition-1")
.withKeySerde(new Serdes.String())
.withValueSerde(new MyClassSerDes()))
.windowedBy(TimeWindows.of(Duration.ofSeconds(5)))
.count(Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("test-store")
.withKeySerde(new Serdes.String())
.withValueSerde(Serdes.Long()));
Here I want to use data from the kTable.
inputstream.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)))
.count(Materialized.<myKey, Long, WindowStore<Bytes, byte[]>>as("str")
.withRetention(Duration.ofMinutes(30)))
.toStream()
.filter((k, v) -> {
// Here get the count for the previous Window.
// Use that count for some computation here.
}
You can add the KTable store to a processor/transformer. For you case, you can replace the filter with flatTransform (or any sibling like transform etc depending if you need access to the key) and connect the store to the operator:
inputstream.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)))
.count(Materialized.<myKey, Long, WindowStore<Bytes, byte[]>>as("str")
.withRetention(Duration.ofMinutes(30))
)
.toStream()
// requires v2.2; otherwise use `transform()`
// if you don't need access to the key, consider to use `flatTransformValues` (v2.3)
.flatTransform(
() -> new Transformer<Windowed<myKey>,
Long,
List<KeyValue<Windowed<myKey>, Long>>() {
private ReadOnlyWindowStore<myKey, Long> store;
public void init(final ProcessorContext context) {
// get a handle on the store by its name
// as specified via `Materialized` above;
// should be read-only
store = (ReadOnlyWindowStore<myKey, Long>)context.getStateStore("str");
}
public List<KeyValue<Windowed<myKey>, Long>> transform(Windowed<myKey> key,
Long value) {
// access `store` as you wish to make a filtering decision
if ( ... ) {
// record passes
return Collection.singletonList(KeyValue.pair(key, value));
} else {
// drop record
return Collection.emptyList();
}
}
public void close() {} // nothing to do
},
"str" // connect the KTable store to the transformer using its name
// as specified via `Materialized` above
);
I am trying to join a stream to 2 differents GlobalTables, treating them as a lookup, more specifically, devices (user agent) and geocoding (ip address).
The issue being with the serialization, but I dont get why. It gets stuck on DEFAULT_VALUE_SERDE_CLASS_CONFIG but the topic to which I want to write is serialized correctly.
//
// Set up serialization / de-serialization
private static Serde<String> stringSerde = Serdes.String();
private static Serde<PodcastData> podcastSerde = StreamsSerdes.PodCastSerde();
private static Serde<GeoCodedData> geocodedSerde = StreamsSerdes.GeoIPSerde();
private static Serde<DeviceData> deviceSerde = StreamsSerdes.DeviceSerde();
private static Serde<JoinedPodcastGeoDeviceData> podcastGeoDeviceSerde = StreamsSerdes.PodcastGeoDeviceSerde();
private static Serde<JoinedPodCastDeviceData> podcastDeviceSerde = StreamsSerdes.PodcastDeviceDataSerde()
...
GlobalKTable<String, DeviceData> deviceIDTable = builder.globalTable(kafkaProperties.getProperty("deviceid-topic"));
GlobalKTable<String, GeoCodedData> geoIPTable = builder.globalTable(kafkaProperties.getProperty("geoip-topic"));
//
// Stream from source topic
KStream<String, PodcastData> podcastStream = builder.stream(
kafkaProperties.getProperty("source-topic"),
Consumed.with(stringSerde, podcastSerde));
//
podcastStream
// left join the podcast stream to the device table, looking up the device
.leftJoin(deviceIDTable,
// get a DeviceData object from the user agent
(podcastID, podcastData) -> podcastData.getUser_agent(),
// join podcast and device and return a JoinedPodCastDeviceData object
(podcastData, deviceData) -> {
JoinedPodCastDeviceData data =
JoinedPodCastDeviceData.builder().build();
data.setPodcastObject(podcastData);
data.setDeviceData(deviceData);
return data;
})
// left join the podcast stream to the geo table, looking up the geo data
.leftJoin(geoIPTable,
// get a Geo object from the ip address
(podcastID, podcastDeviceData) -> podcastDeviceData.getPodcastObject().getIp_address(),
// join podcast and geo
(podcastDeviceData, geoCodedData) -> {
JoinedPodcastGeoDeviceData data=
JoinedPodcastGeoDeviceData.builder().build();
data.setGeoData(geoCodedData);
data.setDeviceData(podcastDeviceData.getDeviceData());
data.setPodcastData(podcastDeviceData.getPodcastObject());
return data;
})
//
.to(kafkaProperties.getProperty("sink-topic"),
Produced.with(stringSerde, podcastGeoDeviceSerde));
...
...
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, stringSerde.getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, stringSerde.getClass().getName());
The error
ERROR java.lang.String cannot be cast to DeviceData
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, stringSerde.getClass().getName());
Due to above value, the application will use String serde as default value serde unless you specify explicitly while making KTable/KStream/GlobalKTable.
Since expected value Type for deviceIDTable is DeviceData, specify that as given below:
You need to define the value serde in GlobalKTable .
GlobalKTable<String, DeviceData> deviceIDTable = builder.globalTable(kafkaProperties.getProperty("deviceid-topic"), Materialized.<String, DeviceData, KeyValueStore<Bytes, byte[]>>as(DEVICE_STORE)
.withKeySerde(stringSerde)
.withValueSerde(deviceSerde));
I have a List of Objects say List<Type1> that I have grouped using type.(using groupingBy)
Now I want to convert that Map> into Type2 that has both the list and the Id of that group.
class Type1{
int id;
int type;
String name;
}
class Type2{
int type;
List<Type1> type1List;
}
This is what I have written to achieve this:
myCustomList
.stream()
.collect(groupingBy(Type1::getType))
.entrySet()
.stream()
.map(type1Item -> new Type2() {
{
setType(type1Item.getKey());
setType1List(type1Item.getValue());
}
})
.collect(Collectors.toList());
This works perfectly. But I am trying to make the code even cleaner. Is there a way to avoid streaming this thing all over again and use some kind of flatmap to achieve this.
You can pass a finisher function to the collectingAndThen to get the work done after the formation of the initial map.
List<Type2> result = myCustomList.stream()
.collect(Collectors.collectingAndThen(Collectors.groupingBy(Type1::getType),
m -> m.entrySet().stream()
.map(e -> new Type2(e.getKey(), e.getValue()))
.collect(Collectors.toList())));
You should give Type2 a constructor of the form
Type2(int type, List<Type1> type1List) {
this.type = type;
this.type1List = type1List;
}
Then, you can write .map(type1Item -> new Type2(type1Item.getKey(), type1Item.getValue())) instead of
.map(type1Item -> new Type2() {
{
setType(type1Item.getKey());
setType1List(type1Item.getValue());
}
})
See also What is Double Brace initialization in Java?
In short, this creates a memory leak, as it creates a subclass of Type2 which captures the type1Item its entire lifetime.
But you can perform the conversion as part of the downstream collector of the groupingBy. This implies that you have to make the toList explicit, to combine it via collectingAndThen with the subsequent mapping:
Collection<Type2> collect = myCustomList
.stream()
.collect(groupingBy(Type1::getType,
collectingAndThen(toList(), l -> new Type2(l.get(0).getType(), l))))
.values();
If you really need a List, you can use
List<Type2> collect = myCustomList
.stream()
.collect(collectingAndThen(groupingBy(Type1::getType,
collectingAndThen(toList(), l -> new Type2(l.get(0).getType(), l))),
m -> new ArrayList<>(m.values())));
You can do as mentioned below:
type1.map( type1Item -> new Type2(
type1Item.getKey(), type1Item
)).collect(Collectors.toList());
I have a class with properties that have their getter and setter each.
I load a list of this class with values from a DB, and I need to create a function that can make a filter over this stream calling different method from the class.
Example:
listActionfilter.stream()
.filter(u -> u.getAccion().toUpperCase().trim().contains(accion))
.collect(Collectors.toList());
I need to do is this:
function xxx('**methodtosearch**', '**valuetosearch**') {
listActionfilter.stream()
.filter(u -> u.('**methodtosearch**')
.toUpperCase().trim().contains('**valuetosearch**'))
.collect(Collectors.toList());
}
Is this possible?
Your function could have the following signature (assuming the object is an ListAction object...
public List<ListAction> function(Predicate<ListAction> predicate) {
return listActionfilter.stream()
.filter(predicate)
.collect(Collectors.toList());
}
And call it the following way
function(u -> u.getAccion().toUpperCase().trim().contains(accion));
Assuming that all your target methods returns String, you can use this :
public List<Action> xxx(Function<Action, String> methodSelector, String valueToMatch) {
return listActionfilter.stream()
.filter(t -> methodSelector.apply(t).toUpperCase().trim(). contains(valueToMatch))
.collect(Collectors.toList());
}
You can invoke the method like this :
List<Action> list1 = xxx(Accion::method1, "value1")
List<Action> list2 = xxx(Accion::method2, "value2")