Currently I have a method in a repository class which fetches data from both a local cache and a remote API.
public Observable<List<Items>> getItemsForUser(String userId {
return Observable.concatArrayEager(
getUserItemsLocal(userId), // returns Observable<List<Items>>
getUserItemsRemote(userId) // returns Observable<List<Items>>
);
}
Currently, the method fetches the local data first (which may be outdated) and returns it, then updates it with the fresh data from the remote API.
I want to change the implementation to use Observable.merge so that if the remote API request completes first, that data gets shown first. However, if I just use Observable.merge I'm concerned that the local database request may return stale data, which will then overwrite the fresh data from the remote.
Basically, I want something like:
public Observable<List<ShoutContent>> getItemsForUser(String userId, ErrorCallback errorCallback) {
return Observable.merge(
getUserItemsRemote(userId),
getUserItemsLocal(userId)
.useOnlyIfFirstResponse()
}
So if the remote API request completes first, then that response is the only one that gets returned. But if the local request completes first, I want to return that, and then return the remote request once it is completed. Does RxJava have anything like this built in?
Edit: I would like to add that getUserItemsRemote does update the local database when the Observable emits, but I don't think that I can ensure that the database will be updated before the local request completes, which leaves the possibility that the local request will respond with stale data.
You can make use of the takeUntil operator.
takeUntil returns an Observable that emits the items emitted by the source Observable until a second ObservableSource emits an item.
In your case, you need to stop observing the local observable, once the remote Observable is emitted. The code is demonstrated below.
public Observable<String> getUserItemsLocal() {
return Observable.just("Local db response")
.delay(5, TimeUnit.SECONDS); // assume local db takes 5 seconds to emit
}
public Observable<String> getUserItemsRemote() {
return Observable.just("Remote Data")
.delay(1, TimeUnit.SECONDS); // remote data comes quicker, in 1 second
}
Your repository code goes like
Observable<String> remoteResponse = getUserItemsRemote();
getUserItemsLocal().takeUntil(remoteResponse)
.mergeWith(remoteResponse)
.subscribe(new Consumer<String>() {
#Override
public void accept(String s) throws Exception {
Log.d(TAG, "result: " + s);
}
});
Related
I have two data sources, each returning a Mono:
class CacheCustomerClient {
Mono<Entity> createCustomer(Customer customer)
}
class MasterCustomerClient {
Mono<Entity> createCustomer(Customer customer)
}
Callers to my application are hitting a Spring WebFlux controller:
#PostMapping
#ResponseStatus(HttpStatus.CREATED)
public Flux<Entity> createCustomer(#RequestBody Customer customer) {
return customerService.createNewCustomer(entity);
}
As long as either data source successfully completes its create operation, I want to immediately return a success response to the caller, however, I still want my service to continue processing the result of the other Mono stream, in the event that an error was encountered, so it can be logged.
The problem seems to be that as soon as a value is returned to the controller, a cancel signal is propagated back through the stream by Spring WebFlux and, thus, no information is logged about a failure.
Here's one attempt:
public Flux<Entity> createCustomer(final Customer customer) {
var cacheCreate = cacheClient
.createCustomer(customer)
.doOnError(WebClientResponseException.class,
err -> log.error("Customer creation failed in cache"));
var masterCreate = masterClient
.createCustomer(customer)
.doOnError(WebClientResponseException.class,
err -> log.error("Customer creation failed in master"));
return Flux.firstWithValue(cacheCreate, masterCreate)
.onErrorMap((err) -> new Exception("Customer creation failed in cache and master"));
}
Flux.firstWithValue() is great for emitting the first non-error value, but then whichever source is lagging behind is cancelled, meaning that any error is never logged out. I've also tried scheduling these two sources on their own Schedulers and that didn't seem to help either.
How can I perform these two calls asynchronously, and emit the first value to the caller, while continuing to listen for emissions on the slower source?
You can achieve that by transforming you operators to "hot" publishers using share() operator:
First subscriber launch the upstream operator, and additional subscribers get back result cached from the first subscriber:
Further Subscriber will share [...] the same result.
Once a second subscription has been done, the publisher is not cancellable:
It's worth noting this is an un-cancellable Subscription.
So, to achieve your requirement:
Apply share() on each of your operators
Launch a subscription on shared publishers to trigger processing
Use shared operators in your pipeline (here firstWithValue).
Sample example:
import java.time.Duration;
import reactor.core.publisher.Mono;
public class TestUncancellableMono {
// Mock a mono successing quickly
static Mono<String> quickSuccess() {
return Mono.delay(Duration.ofMillis(200)).thenReturn("SUCCESS !");
}
// Mock a mono taking more time and ending in error.
static Mono<String> longError() {
return Mono.delay(Duration.ofSeconds(1))
.<String>then(Mono.error(new Exception("ERROR !")))
.doOnCancel(() -> System.out.println("CANCELLED"))
.doOnError(err -> System.out.println(err.getMessage()));
}
public static void main(String[] args) throws Exception {
// Transform to hot publisher
var sharedQuick = quickSuccess().share();
var sharedLong = longError().share();
// Trigger launch
sharedQuick.subscribe();
sharedLong.subscribe();
// Subscribe back to get the cached result
Mono
.firstWithValue(sharedQuick, sharedLong)
.subscribe(System.out::println, err -> System.out.println(err.getMessage()));
// Wait for subscription to end.
Thread.sleep(2000);
}
}
The output of the sample is:
SUCCESS !
ERROR !
We can see that error message has been propagated properly, and that upstream publisher has not been cancelled.
Trying to replace the using callback with coroutines. Having a implementation using callback and not sure what is the right approach if coroutines could help.
This is the implementation with callback.
It has a class repository to provide data from either local database or network remote.
class Repository() {
var callback = //callback is provided by the caller
var isReady = false
var data = null
var firstimeCall = true //only need to get from database at first time call
fun getData(cb: ICallback) {
callback = cb
isReady = false
if (firstimeCall) {
firstimeCall = false
data = doGetDataFromDatabase() //sync call to get data from database
isReady = true
callback.onComplete(this)
}
isReady = false
data = doGetDataFromNetwork() {// async call and lamda as the callback
isReady = true
saveToDatabase(data)
callback.onComplete(this)
}
}
}
the repository.getData() could be called multiple times, only first time it will return the data from database first, then
getting from network and saving, then call callback.onComplete() to return the data.
Any other time, it will only do getting from network/save/return data through the callback.
the use case are:
directly using Repository, like
repository.getData() -- 1st time call
repository.getData() -- later call it again
there are multiple repositories, the data from each one will be aggregated into a final data.
for this case there is a Aggregator to hold the repositories, and provides onComplete() callback to process data if all
repositories are ready.
class Aggregator {
var repos = ArrayList<Repository>()
fun getData() {
for (r in repos) {
Thread {
r.getData(callback)
}.start()
}
}
fun processData(data: ArrayList()) {
......
}
val callback = object ICallback (onComplete{repo->
val hasAllComplete = repos.all {
it.isReady
}
if (hasAllComplete) {
var finalData = ArrayList<Data>()
for (r in repos) {
finalData.add(r.data)
}
processData(finalData)
}
})
}
so in the case it has two Repository, the Aggregator.getData() will get data from the two repositories.
when one Repository is complete its getData() call, it will callback to the callback's onComplete() where
the Aggregator will check wether all repositories are ready for data to be processed.
The same callback flow is used for the network call aswell.
Question:
In this case how to change to use coroutines, so that only after getting data from the database are complete for both repositories,
then it will start to get data from the network, without using callbacks.
I'm not sure if it's relevant anymore, but you could have a look at callbackFlow.
More info here:
https://medium.com/#elizarov/callbacks-and-kotlin-flows-2b53aa2525cf#1aaf
I have a similar problem, and I think this might be the solution to it.
Make sure you also read more about Flow and its usage before actually using it, since there are some caveats with handling exceptions (exception transparency), etc.
I'm using rxjs with NodeJS in backend.
I have a Rest API which allow consumers to run remote yarn installation process. The install function returns an observable of the process. So when the module is installed successfully it emits a value in the observable and complete. At this point, the Rest API will returns a response to the user to say that the installation is successful. In case that the installation fails, the process will throw an Error in the stream and the Rest API returns another response with the error information.
My issue is:
The API is called multiple times in parallel by consumers, so there will be a parallel installations in the backend.
I tried throttle operator to create a queue but it keeps the first stream active. So if the first process is "completed", it returns "true" but the stream doesn't complete
export class MyService {
// the function called by the REST API
installGlobal(moduleName: string): Observable < boolean > {
// I think, there are something to do here to make it queuing
return this.run('yarn', ['global', 'add', moduleName]);
}
private run(cmd: string, args: string[]): Observable < boolean > {
const cmd$ = fromPromise(spawn(cmd, args)).pipe(
map(stdout => {
this.logger.info(`Install Module Successfully`);
this.logger.info(`stdout: ${stdout.toString()}`);
return true;
}),
catchError(error => {
const errorMessage: string = error.stderr.toString();
return _throw(errorMessage.substr(errorMessage.indexOf(' ') + 1));
})
);
return cmd$;
}
}
My expectation:
Either there are multiple request, they must be queued. So the first one will be treated and all parallel onces must be queued. When the first is processed, it must returns the response to the API consumers (like 200 completed) and resume the next stream from the queue.
[UPDATE-01 July 2019]: adding an example
You can fetch a demo of the code at stackblitz
I have reimplemented the existant code and i'm simulating my API call by subscribing multi time to the service which will call the queue
A simple queque in Rxjs can be done like below
const queque=new Subject()
// sequential processing
queue.pipe(concatMap(item=>yourObservableFunction(item)).subscribe()
// add stuff to the queue
queque.next(item)
I have an endpoint streamed as in the sample code block. When streaming, I call an async method through streamHelper.getStreamSuspendCount(). I am stopping this async method in changing state. But I can not access this async method when the browser is closed and the session is terminated. I am stopping the async method in session scope when changing state. But I can not access this async method when the browser is closed and the session is terminated. How can I access this scope when Session is closed?
#RequestMapping(value = "/stream/{columnId}/suspendCount", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
#ResponseBody
public Flux<Integer> suspendCount(#PathVariable String columnId) {
ColumnObject columnObject = streamHelper.findColumnObjectInListById(columnId);
return streamHelper.getStreamSuspendCount(columnObject);
}
getStreamSuspendCount(ColumnObject columnObject) {
...
//async flux
Flux<?> newFlux = beSubscribeFlow.get(i);
Disposable disposable = newFlux.subscribe();
beDisposeFlow.add(disposable); // my session scope variable. if change state, i will kill disposable (dispose()).
...
return Flux.fromStream(Stream.generate(() -> columnObject.getPendingObject().size())).distinctUntilChanged()
.doOnNext(i -> {
System.out.println(i);
}));
}
I think part of the problem is that you are attempting to get a Disposable that you want to call at the end of the session. But in doing so, you are subscribing to the sequence yourself. Spring Framework will also subscribe to the Flux returned by getStreamSuspendCount, and it is THAT subscription that needs to be cancelled for the SSE client to get notified.
Now how to achieve this? What you need is a sort of "valve" that will cancel its source upon receiving an external signal. This is what takeUntilOther(Publisher<?>) does.
So now you need a Publisher<?> that you can tie to the session lifecycle (more specifically the session close event): as soon as it emits, takeUntilOther will cancel its source.
2 options there:
the session close event is exposed in a listener-like API: use Mono.create
you really need to manually trigger the cancel: use MonoProcessor.create() and when the time comes, push any value through it
Here are simplified examples with made up APIs to clarify:
Create
return theFluxForSSE.takeUntilOther(Mono.create(sink ->
sessionEvent.registerListenerForClose(closeEvent -> sink.success(closeEvent))
));
MonoProcessor
MonoProcessor<String> processor = MonoProcessor.create();
beDisposeFlow.add(processor); // make it available to your session scope?
return theFluxForSSE.takeUntilOther(processor); //Spring will subscribe to this
Let's simulate the session close with a scheduled task:
Executors.newSingleThreadScheduledExecutor().schedule(() ->
processor.onNext("STOP") // that's the key part: manually sending data through the processor to signal takeUntilOther
, 2, TimeUnit.SECONDS);
Here is a simulated unit test example that you can run to better understand what happens:
#Test
public void simulation() {
Flux<Long> theFluxForSSE = Flux.interval(Duration.ofMillis(100));
MonoProcessor<String> processor = MonoProcessor.create();
Executors.newSingleThreadScheduledExecutor().schedule(() -> processor.onNext("STOP"), 2, TimeUnit.SECONDS);
theFluxForSSE.takeUntilOther(processor.log())
.log()
.blockLast();
}
I have a list of promises that needs to be executed in parallel and in an asynchronous manner.Say,i have,
List<Promise<X>> list;
Once all the parallel request completes, i need to make another request say "Y". Here is my GWT code,
GQuery.when(list).done(...).fail(..)
But the above doesn seem to work!.How can i pass a list of promises to GQuery?.Is the above synctax valid?.
If you create a sample GWT project in Eclipse, a simple asynchronous RPC call is created. You can take that as a template to change it the way you need. With the callback of the request is it possible to display your "Y".
// Set up the callback object.
AsyncCallback<List<Promise<X>>> callback = new AsyncCallback<List<Promise<X>>>() {
public void onFailure(Throwable caught) {
// TODO: Do something with errors.
}
public void onSuccess(List<Promise<X>> result) {
// TODO: DO something with the result.
}
};
You should also read the documentations, at least...
http://www.gwtproject.org/doc/latest/tutorial/RPC.html