RxJava Caching Results of Network Call IDs from a Stream (Redis or similar caching solution) - caching

Question

I need some help with RxJava. I have an expensive network call which returns an Observable (stream of Adverts from elasticsearch). I want to cache the ID property of each emitted item (Advert) for 10 minutes (in Redis), so that subsequent calls in the following 10 minutes use the IDs from Cache to fetch the adverts from Elasticsearch.
I've got some code - which goes some way towards achieving the desired outcome, (credit to following blog ..
http://blog.danlew.net/2015/06/22/loading-data-from-multiple-sources-with-rxjava/)
It can cache each emitted item from the stream, what I need is it to cache all the IDs from those items in the stream as 1 cache entry
Code so far is here https://github.com/tonymurphy/rxjava for anyone interested, snippets below
#Component
public class CachingObservable {
private final Logger logger = LoggerFactory.getLogger(CachingObservable.class);
#Autowired
private AdvertService advertService;
// Each "network" response is different
private Long requestNumber = 0L;
public Observable<Advert> getAdverts(final String location) {
Observable<Advert> memory = memory(location);
Observable<Advert> network = network(location);
Observable<Advert> networkWithSave = network.doOnNext(new Action1<Advert>() {
#Override
public void call(Advert advert) {
List<Long> ids = new ArrayList<Long>();
ids.add(advert.getId());
advertService.cache(location, ids);
}
});
// Retrieve the first source with data - concat checks in order
Observable<Advert> source = Observable.concat(memory,
networkWithSave)
.first();
return source;
}
From my understanding, the concat method is not really useful for my use case. I need to know if/when the network observable completes, I need to get the list of advert id's returned and I need to store them in the cache. I could subscribe to the network observable - but I want this to be lazy - only called if no data is found in the cache. So the following updated code doesn't work.. any ideas appreciated
public Observable<Advert> getAdverts(final String location) {
Observable<Advert> memory = memory(location);
Observable<Advert> network = network(location);
Observable<Advert> networkWithSave = network.doOnNext(new Action1<Advert>() {
#Override
public void call(Advert advert) {
List<Long> ids = new ArrayList<Long>();
ids.add(advert.getId());
advertService.cache(location, ids);
}
});
// Retrieve the first source with data - concat checks in order
Observable<Advert> source = Observable.concat(memory,
networkWithSave)
.first();
Observable<List<Advert>> listObservable = networkWithSave.toList();
final Func1<List<Advert>, List<Long>> transformer = new Func1<List<Advert>, List<Long>>() {
#Override
public List<Long> call(List<Advert> adverts) {
List<Long> ids = new ArrayList<Long>();
for (Advert advert : adverts) {
ids.add(advert.getId());
}
return ids;
}
};
listObservable.map(transformer).subscribe(new Action1<List<Long>>() {
#Override
public void call(List<Long> ids) {
logger.info("ids {}", ids);
}
});
return source;
}

Answer 1

What I would do is use filter to make sure the old content of the cache is not emitted so the concat jumps to the network call:
Subject<Pair<Long, List<Advert>>, Pair<Long, List<Advert>>> cache =
BehaviorSubject.create().toSerialized();
static final long RETENTION_TIME = 10L * 60 * 1000;
Observable<Advert> memory = cache.filter(v ->
v.first + RETENTION_TIME > System.currentTimeMillis()).flatMapIterable(v -> v);
Observable<Advert> network = ...
Observable<Advert> networkWithSave = network.toList().doOnNext(v ->
cache.onNext(Pair.of(System.currentTimeMillis(), v)).flatMapIterable(v -> v)
);
return memory.switchIfEmpty(network);

Answer 2

Ok, I think I have a solution which will work for me. I may have overlooked something, but it should be easy?
public Observable<Advert> getAdverts(final String location) {
Observable<Advert> memory = memory(location);
final Observable<Advert> network = network(location);
final Func1<List<Advert>, List<Long>> advertToIdTransformer = convertAdvertsToIds();
memory.isEmpty().subscribe(new Action1<Boolean>() {
#Override
public void call(Boolean aBoolean) {
if (aBoolean.equals(Boolean.TRUE)) {
Observable<List<Long>> listObservable = network.toList().map(advertToIdTransformer);
listObservable.subscribe(new Action1<List<Long>>() {
#Override
public void call(List<Long> ids) {
logger.info("Caching ids {}", ids);
advertService.cache(location, ids);
}
});
}
}
});
// Retrieve the first source with data - concat checks in order
Observable<Advert> source = Observable.concat(memory,
network)
.first();
return source;
}