Have multiple operations on a single event - rxjs

Trying to have a event triggering multiple switchMap with the initial event data.
Each actions creates a promise to some transform that is then written in the file system. The actions are independent and unrelated, but uses the same data, just for different purpose, so they should not be merged.
Currently using taps instead of switchMap, that can lead to multiple event running the at the same time.
const SomeApiCall = () => {return {some: 'data'} }
const AllowsDoAction = () => {console.log('Parsing API and writting some things to FS -- PLACEHOLDER')}
const SomeTimeDoThisActionTo = () => {console.log('Parsing API and writting some other things to fs, by asking more data from the API and first, so it is long thing to do, so a new event can arrive first -- PLACEHOLDER')}
const deepEqual = (prev, cur) => prev === cur // normally a proper deepEqual...
const taps = [tap(AllowsDoAction)];
if (someCondition) taps.push(SomeTimeDoThisActionTo)
const observable = timer(0, 500).pipe(
exhaustMap(SomeApiCall),
distinctUntilChanged((prev, cur) => deepEqual(prev, cur))
...taps
);

I would return the filesystem write observables (edit 2)
const SomeApiCall = () => {
return of({ some: 'data' });
};
const AllowsDoAction = () => {
console.log('Parsing API and writting some things to FS -- PLACEHOLDER');
return timer(100).pipe(map(() => 'write 1 finished'));
};
const SomeTimeDoThisActionTo = () => {
console.log(
'Parsing API and writting some other things to fs, by asking more data from the API and first, so it is long thing to do, so a new event can arrive first -- PLACEHOLDER'
);
return timer(1000).pipe(map(() => 'write 2 finished'));
};
And then use concatMap to wait for all filesystem operation to complete.
const deepEqual = (prev, cur) => prev === cur; // normally a proper deepEqual...
const taps = [(AllowsDoAction)];
const someCondition = true;
if (someCondition) {
taps.push(SomeTimeDoThisActionTo);
}
const reduxStorageEvent$ = of('replace this with real event');
const observable = merge(timer(0, 500), reduxStorageEvent$).pipe(
exhaustMap(SomeApiCall),
distinctUntilChanged((prev, cur) => deepEqual(prev, cur)),
// use switchMap to cancel previous writes (edit 2)
// await latest write operations, before starting new writes
concatMap((someData) => {
const writes = taps.map((tapFx) => {
return tapFx(someData);
});
// wait for all writes
return forkJoin(...writes);
})
);
concatMap is like a queue. The first one in this queue has to finish before the second one can start.

Related

Is having more than one API fetch request in a React useEffect hook always very slow?

I am writing a simple application using React to fetch and display data from the Star Wars API. I first fetch information about a particular planet. The response JSON for a given planet contains a bunch of data, including an array of URLs pointing to further data about notable residents of said planet. I next call each of those URLs in order to display a list of the names of the residents of the current planet.
This code works, but is slow as heck:
const url = `https://swapi.dev/api/planets/`;
const [currentPlanetNumber, setCurrentPlanetNumber] = React.useState(1);
const [currentPlanet, setCurrentPlanet] = React.useState({});
const [currentPlanetResidentsDetails, setCurrentPlanetResidentsDetails] =
React.useState([]);
React.useEffect(() => {
(async () => {
const planetData = await fetch(`${url}${currentPlanetNumber}/`).then(
(response) => response.json()
);
setCurrentPlanet(planetData);
if (planetData.residents.length === 0) {
setCurrentPlanetResidentsDetails(["No notable residents"]);
} else {
const residentsURLs = planetData.residents;
const residentsNames = await Promise.all(
residentsURLs.map(async (item) => {
const name = await fetch(item).then((response) => response.json());
const newName = name.name;
return newName;
})
);
setCurrentPlanetResidentsDetails(residentsNames);
}
})();
}, [currentPlanetNumber]);
The following code works fairly fast for this:
const url = `https://swapi.dev/api/planets/`;
const [currentPlanetNumber, setCurrentPlanetNumber] = React.useState(1);
const [currentPlanet, setCurrentPlanet] = React.useState({});
const [currentPlanetResidentsDetails, setCurrentPlanetResidentsDetails] =
React.useState([]);
React.useEffect(() => {
(async () => {
const planetData = await fetch(`${url}${currentPlanetNumber}/`).then(
(response) => response.json()
);
setCurrentPlanet(planetData);
})();
}, [currentPlanetNumber]);
React.useEffect(() => {
(async () => {
if (currentPlanet.residents.length === 0) {
setCurrentPlanetResidentsDetails(["No notable residents"]);
} else {
const residentsURLs = currentPlanet.residents;
const residentsNames = await Promise.all(
residentsURLs.map(async (item) => {
const name = await fetch(item).then((response) => response.json());
const newName = name.name;
return newName;
})
);
setCurrentPlanetResidentsDetails(residentsNames);
}
})();
}, [currentPlanet]);
What makes the second one so much faster? I assumed that they would both take about the same length of time, because the same number of fetch requests get done either way.
Is it a good rule of thumb to not have more than one fetch request an any given useEffect hook?
No, there is no rule of thumb stating not to have more than one fetch request in a given useEffect.
In your first example, the fetch requests are fired consecutively, while in the second example they are fired concurrently.
Your first example seems to be more appropriate than the second. In the second example, it seems to you that the code is executing faster because both effects are firing concurrently when the component mounts. On subsequent changes to 'currentPlanetNumber', both examples should execute in the same amount of time.

Delay batch of observables with RxJS

I perform http requests to my db and have noticed that if I send all the requests at once, some of them will get a timeout errors. I'd like to add a delay between calls so the server doesn't get overloaded. I'm trying to find the RxJS solution to this problem and don't want to add a setTimeout.
Here is what I currently do:
let observables = [];
for(let int = 0; int < 10000; int++){
observables.push(new Observable((observer) => {
db.add(doc[int], (err, result)=>{
observer.next();
observer.complete();
})
}))
}
forkJoin(observables).subscribe(
data => {
},
error => {
console.log(error);
},
() => {
db.close();
}
);
You can indeed achieve this with Rxjs quite nicely. You'll need higher order observables, which means you'll emit an observable into an observable, and the higher order observable will flatten this out for you.
The nice thing about this approach is that you can easily run X requests in // without having to manage the pool of requests yourself.
Here's the working code:
import { Observable, Subject } from "rxjs";
import { mergeAll, take, tap } from "rxjs/operators";
// this is just a mock to demonstrate how it'd behave if the API was
// taking 2s to reply for a call
const mockDbAddHtppCall = (id, cb) =>
setTimeout(() => {
cb(null, `some result for call "${id}"`);
}, 2000);
// I have no idea what your response type looks like so I'm assigning
// any but of course you should have your own type instead of this
type YourRequestType = any;
const NUMBER_OF_ITEMS_TO_FETCH = 10;
const calls$$ = new Subject<Observable<YourRequestType>>();
calls$$
.pipe(
mergeAll(3),
take(NUMBER_OF_ITEMS_TO_FETCH),
tap({ complete: () => console.log(`All calls are done`) })
)
.subscribe(console.log);
for (let id = 0; id < NUMBER_OF_ITEMS_TO_FETCH; id++) {
calls$$.next(
new Observable(observer => {
console.log(`Starting a request for ID "${id}""`);
mockDbAddHtppCall(id, (err, result) => {
if (err) {
observer.error(err);
} else {
observer.next(result);
observer.complete();
}
});
})
);
}
And a live demo on Stackblitz: https://stackblitz.com/edit/rxjs-z1x5m9
Please open the console of your browser and note that the console log showing when a call is being triggered starts straight away for 3 of them, and then wait for 1 to finish before picking up another one.
Looks like you could use an initial timer to trigger the http calls. e.g.
timer(delayTime).pipe(combineLatest(()=>sendHttpRequest()));
This would only trigger the sendHttpRequest() method after the timer observable had completed.
So with your solution. You could do the following...
observables.push(
timer(delay + int).pipe(combineLatest(new Observable((observer) => {
db.add(doc[int], (err, result)=>{
observer.next();
observer.complete();
}))
}))
Where delay could probably start off at 0 and you could increase it using the int index of your loop by some margin.
Timer docs: https://www.learnrxjs.io/learn-rxjs/operators/creation/timer
Combine latest docs: https://www.learnrxjs.io/learn-rxjs/operators/combination/combinelatest
merge with concurrent value:
mergeAll and mergeMap both allow you to define the max number of subscribed observables. mergeAll(1)/mergeMap(LAMBDA, 1) is basically concatAll()/concatMap(LAMBDA).
merge is basically just the static mergeAll
Here's how you might use that:
let observables = [...Array(10000).keys()].map(intV =>
new Observable(observer => {
db.add(doc[intV], (err, result) => {
observer.next();
observer.complete();
});
})
);
const MAX_CONCURRENT_REQUESTS = 10;
merge(...observables, MAX_CONCURRENT_REQUESTS).subscribe({
next: data => {},
error: err => console.log(err),
complete: () => db.close()
});
Of note: This doesn't batch your calls, but it should solve the problem described and it may be a bit faster than batching as well.
mergeMap with concurrent value:
Perhaps a slightly more RxJS way using range and mergeMap
const MAX_CONCURRENT_REQUESTS = 10;
range(0, 10000).pipe(
mergeMap(intV =>
new Observable(observer => {
db.add(doc[intV], (err, result) => {
observer.next();
observer.complete();
});
}),
MAX_CONCURRENT_REQUESTS
)
).subscribe({
next: data => {},
error: err => console.log(err),
complete: () => db.close()
});

Multiple subscriptions nested into one subscription

I find myself puzzled trying to set a very simple rxjs flow of subscriptions. Having multiple non-related subscriptions nested into another.
I'm in an angular application and I need a subject to be filled with next before doing other subscriptions.
Here would be the nested version of what I want to achieve.
subject0.subscribe(a => {
this.a = a;
subject1.subscribe(x => {
// Do some stuff that require this.a to exists
});
subject2.subscribe(y => {
// Do some stuff that require this.a to exists
});
});
I know that nested subscriptions are not good practice, I tried using flatMap or concatMap but didn't really get how to realize this.
It's always a good idea to separate the data streams per Observable so you can easily combine them later on.
const first$ = this.http.get('one').pipe(
shareReplay(1)
)
The shareReplay is used to make the Observable hot so it won't call http.get('one') per each subscription.
const second$ = this.first$.pipe(
flatMap(firstCallResult => this.http.post('second', firstCallResult))
);
const third$ = this.first$.pipe(
flatMap(firstCallResult => this.http.post('third', firstCallResult))
);
After this you can perform subscriptions to the Observables you need:
second$.subscribe(()=>{}) // in this case two requests will be sent - the first one (if there were no subscribes before) and the second one
third$.subscribe(() => {}) // only one request is sent - the first$ already has the response cached
If you do not want to store the first$'s value anywhere, simply transform this to:
this.http.get('one').pipe(
flatMap(firstCallResult => combineLatest([
this.http.post('two', firstCallResult),
this.http.post('three', firstCallResult)
])
).subscribe(([secondCallResult, thirdCallResult]) => {})
Also you can use BehaviorSubject that stores the value in it:
const behaviorSubject = new BehaviorSubject<string>(null); // using BehaviorSubject does not require you to subscribe to it (because it's a hot Observable)
const first$ = behaviorSubject.pipe(
filter(Boolean), // to avoid emitting null at the beginning
flatMap(subjectValue => this.http.get('one?' + subjectValue))
)
const second$ = first$.pipe(
flatMap(firstRes => this.http.post('two', firstRes))
)
const third$ = first$.pipe(
flatMap(()=>{...})
)
behaviorSubject.next('1') // second$ and third$ will emit new values
behaviorSubject.next('2') // second$ and third$ will emit the updated values again
You can do that using the concat operator.
const first = of('first').pipe(tap((value) => { /* doSomething */ }));
const second = of('second').pipe(tap((value) => { /* doSomething */ }));
const third = of('third').pipe(tap((value) => { /* doSomething */ }));
concat(first, second, third).subscribe();
This way, everything is chained and executed in the same order as defined.
EDIT
const first = of('first').pipe(tap(value => {
// doSomething
combineLatest(second, third).subscribe();
}));
const second = of('second').pipe(tap(value => { /* doSomething */ }));
const third = of('third').pipe(tap(value => { /* doSomething */ }));
first.subscribe();
This way, second and third are running asynchronously as soon as first emits.
You could do something like this:
subject$: Subject<any> = new Subject();
this.subject$.pipe(
switchMap(() => subject0),
tap(a => {
this.a = a;
}),
switchMap(() => subject1),
tap(x => {
// Do some stuff that require this.a to exists
}),
switchMap(() => subject2),
tap(y => {
// Do some stuff that require this.a to exists
})
);
if you want to trigger this, simply call this.subject$.next();
EDIT:
Here is an possible approach with forkJoin, that shout call the subjects parallel.
subject$: Subject<any> = new Subject();
this.subject$.pipe(
switchMap(() => subject0),
tap(a => {
this.a = a;
}),
switchMap(
() => forkJoin(
subject1,
subject2
)),
tap([x,y] => {
// Do some stuff that require this.a to exists
})
);

Do something if RxJs subject's refCount drops to zero

I'm working on a service layer that manages subscriptions.
I provide subject-backed observables to consumers like this:
const subject = new Subject();
_trackedSubjects.push(subject);
return subject.asObservable();
Different consumers may monitor the channel, so there may be several observables attached to each subject.
I'd like to monitor the count of subject.observers and if it ever drops back to 0, do some cleanup in my library.
I have looked at refCount, but this only is available on Observable.
I'd love to find something like:
subject.onObserverCountChange((cur, prev) =>
if(cur === 0 && prev !== 0) { cleanUp(subject) }
)
Is there a way to automatic cleanup like this on a subject?
Instead of using Subject - you should probably describe setup/cleanup logic when creating observable. See the example:
const { Observable } = rxjs; // = require("rxjs")
const { share } = rxjs.operators; // = require("rxjs/operators")
const eventSource$ = Observable.create(o => {
console.log('setup');
let i = 0
const interval = setInterval(
() => o.next(i++),
1000
);
return () => {
console.log('cleanup');
clearInterval(interval);
}
});
const events$ = eventSource$.pipe(share());
const first = events$.subscribe(e => console.log('first: ', e));
const second = events$.subscribe(e => console.log('second: ', e));
setTimeout(() => first.unsubscribe(), 3000);
setTimeout(() => second.unsubscribe(), 5000);
<script src="https://unpkg.com/rxjs#6.2.2/bundles/rxjs.umd.min.js"></script>

rxjs5 merge and error handling

I would like to combine/merge multiple Observables and when each of them is completed execute a finally function. The merge operator seems to execute each subscription in parallel, which is what I need, but if any of them throws an error the execution is halted.
RxJS version 4 has an operator mergeDelayError that should keep the all subscriptions executing till all of them are completed, but this operator isn't implemented in version 5.
Should I revert to a different operator?
var source1 = Rx.Observable.of(1,2,3).delay(3000);
var source2 = Rx.Observable.throw(new Error('woops'));
var source3 = Rx.Observable.of(4,5,6).delay(1000);
// Combine the 3 sources into 1
var source = Rx.Observable
.merge(source1, source2, source3)
.finally(() => {
// finally is executed before all
// subscriptions are completed.
console.log('finally');
});
var subscription = source.subscribe(
x => console.log('next:', x),
e => console.log('error:', e),
() => console.log('completed'));
JSBin
We can avoid blocking the stream by collecting the errors and emitting them at the end.
function mergeDelayError(...sources) {
const errors = [];
const catching = sources.map(obs => obs.catch(e => {
errors.push(e);
return Rx.Observable.empty();
}));
return Rx.Observable
.merge(...catching)
.concat(Rx.Observable.defer(
() => errors.length === 0 ? Rx.Observable.empty() : Rx.Observable.throw(errors)));
}
const source1 = Rx.Observable.of(1,2,3);
const source2 = Rx.Observable.throw(new Error('woops'));
const source3 = Rx.Observable.of(4,5,6);
mergeDelayError(source1, source2, source3).subscribe(
x => console.log('next:', x),
e => console.log('error:', e),
() => console.log('completed'));
I think you can simulate the same behavior by using catch(). You'll just need to append it to every source Observable:
const sources = [source1, source2, source3].map(obs =>
obs.catch(() => Observable.empty())
);
Rx.Observable
.merge(sources)
.finally(...)
...
If you don't want to swallow your errors, but want to delay them to the end, you can:
const mergeDelayErrors = [];
const sources = [source1, source2, source3].map(obs => obs.catch((error) => {
mergeDelayErrors.push(error);
return Rx.Observable.empty();
}));
return Rx.Observable
.merge(...sources)
.toArray()
.flatMap(allEmissions => {
let spreadObs = Rx.Observable.of(...allEmissions);
if (mergeDelayErrors.length) {
spreadObs = spreadObs.concat(Rx.Observable.throw(mergeDelayErrors));
}
return spreadObs;
})
You may want to throw only the first error, or create a CompositeError. I'm not sure how mergeDelayErrors originally behaved when multiple errors were thrown.
Unfortunately, because this implementation must wait until the all observables complete before emitting errors, it also waits until all observables complete before emitting next. This is likely not the original behavior of mergeDelayError, which is supposed to emit as a stream, rather than emitting them all at the end.

Resources