Rxjs Interval switchMap once http call finishes - rxjs

We have a timer that ticks e.g. every 2 seconds, whenever there is a tick we make a http call. The response time varies, it might be 100 ms but it might be 4 seconds. The catch is that we only make a new http call if the previous one completed. So we have 2 cases:
#1 If each http call takes 10sec we should be only sending a request every 10 seconds.
#2 If each http call takes 100ms we should be sending a request every 2 seconds.
What is also difficult in this scenario is that the getHttp call relies on value from the interval$ stream.
The best solution I came up so far is this:
import { of, interval, timer, combineLatest, BehaviorSubject } from 'rxjs';
import { filter, map, switchMap, tap } from 'rxjs/operators';
const interval$ = interval(2000).pipe(tap(console.log))
const readyForNext$ = new BehaviorSubject(true);
// Mocked http request that (in reality) uses the value from interval$
function getHttp(value) {
return of(value).pipe(
tap(() => readyForNext$.next(false)),
tap(() => console.log('getHttp: starting ')),
switchMap(() => timer(3000)),
tap(() => console.log('getHttp: resolve')),
map(() => 'hello'),
);
}
const anotherMockedStream$ = of(1) // In real code we have 2 streams in combineLatest
combineLatest([interval$, readyForNext$])
.pipe(
filter(([_, isReady]) => isReady),
switchMap(([value]) => {
return combineLatest([getHttp(value), anotherMockedStream$])
}),
tap(() => readyForNext$.next(true)),
)
.subscribe(console.log);
Playground: https://playcode.io/1023332
Alternative playground: https://stackblitz.com/edit/typescript-jdjjsc?file=index.ts
There are however two problem with this solution, it only works for case #1 and it is not very elegant (the use of readyForNext$). How to improve this code?

Thanks to #martin for pointing to exhaustMap, this is the correct operator for the operation, it seems to me that it is similar to switchMap but only does the switch once the steam is completed (or the emitted value reaches the subscription).
The working code looks as follows:
import { of, interval, timer, combineLatest } from 'rxjs';
import { map, switchMap, tap, exhaustMap } from 'rxjs/operators';
const interval$ = interval(2000).pipe(tap(console.log))
// Mocked http request that (in reality) uses the value from interval$
function getHttp(value) {
return of(value).pipe(
tap(() => console.log('getHttp: starting ')),
switchMap(() => timer(3000)),
tap(() => console.log('getHttp: resolve')),
map(() => 'hello'),
);
}
const anotherMockedStream$ = of(1)
interval$.pipe(
exhaustMap(val => {
return combineLatest([getHttp(val), anotherMockedStream$])
})
).subscribe(console.log);
With it I was able to remove the unnecessary readyForNext$ and it also works for both case #1 and case #2, i.e. if timer's value is lower or higher than interval.

Related

Can I remove .subscribe() from map by using another RxJS operator?

Is there an alternative to doing .subscribe() in map?
return this.http.patch<any>(URL, { test: true }).pipe(
tap(_ => this.nextSomething$.next({})),
filter(_ => this.filter),
map(resp => {
this.someObservable({ message: 'do' }).subscribe()
return resp
})
)
I tried doing switchMap and returning the previous response but my observable does not complete.
switchMap(prevResp =>
this.someObservable({ message: 'do' }) }).pipe(map( _ => prevResp))
)
Thank you
If your observable is not completing when you switch to using the flattening operator then that means the target observable in that operator isn't completing. So you might want to check what's going on in someObservable that causes it to never complete. It is likey that behavior isn't desirable.
If you know that someObservable will emit at least once, then you can add the first operator to the inner pipe method:
switchMap(prevResp =>
this.someObservable({ message: 'do' }) }).pipe(
first(),
map( _ => prevResp)
)
)
If you don't care what someObservable does - you don't want to wait for it just want it to execute, then wrap the observable in firstValueFrom. It converts an observable into a promise that emits the first result. This code smells, but it should do the trick.
this.http.patch<any>(URL, { test: true }).pipe(
tap(() => this.nextSomething$.next({})),
filter(() => this.filter),
tap(() => firstValueFrom(this.someObservable({ message: 'do' })))
)
Maybe, you don't care to wait for this observable to emit, but you're still looking for an orderly execution of someObservable. In that case you can use a subject that emits every time you want it called, and use concatMap to ensure the execution is performed in an orderly fashion.
private readonly someObservableRequest = new Subject<string>();
constructor() {
this.someObservableRequest.pipe(
concatMap((message) => this.someObservable({ message }))
).subscribe(); // be kind, please unsubscribe
}
someMethod() {
return this.http.patch<any>(URL, { test: true }).pipe(
tap(_ => this.nextSomething$.next({})),
filter(_ => this.filter),
tap(() => this.someObservableRequest('do'))
);
}
If I understand your point right, you have to execute consecutively 2 Observables.
In this case you need yo use one so called "higher order" operators, i.e. either concatMap or switchMap (there are other "higher order" operators but I feel they do not apply to your case).
The code then would look like this
myNewOservable this.http.patch<any>(URL, { test: true }).pipe(
tap(_ => this.nextSomething$.next({})),
filter(_ => this.filter),
// an higher order operator returns an Observable
concatMap(resp => {
return this.someObservable({ message: 'do' })
})
)
return myNewOservable
Now you can subscribe to myNewOservable.
In my example I have used concatMap which ensures that a value notified by the upstream Observable is processed through the downstream pipeline before processing the next value from upstream.
I could have used also switchMap, which has a slightly different behavior: as soon as a new value is notified by the upstream Observable, any downstream processing is terminated.
In this case, given that http emits only one value and then completes the 2 operators act the same.

How to poll several endpoints sequentially with rxjs?

I am trying to achieve the following with Rxjs: given an array of job ids, for every id in the array, poll an endpoint that returns the status of the job. The status can be either "RUNNING", or "FINISHED". The code should poll jobs one after the other, and continue the polling until the jobs are in the "RUNNING" status. As soon as a job reaches the "FINISHED" status, it should be passed downstream, and excluded from further polling.
Below is a minimal toy case that demonstrates the problem.
const {
from,
of,
interval,
mergeMap,
filter,
take,
tap,
delay
} = rxjs;
const { range } = _;
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input })
.pipe(delay(500));
};
const number$ = from(range(1, 10));
const poll = (number) => interval(5000).pipe(
mergeMap(() => {
return doRequest(number)
}),
tap(console.log),
filter(( {status} ) => status === 'FINISHED'),
take(1)
);
const printout$ = number$.pipe(
mergeMap((number) => {
return poll(number)
})
);
printout$.subscribe(console.log);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.21/lodash.min.js"></script>
It does most of what I described; but it polls all endpoints simultaneously rather than one after another. Here, roughly, is the pattern I would like to achieve:
starting with ids: [1, 2, 3]
polling: await request 1 then await request 2 then await request 3
then wait for n seconds; then repeat
after job 2 is finished, send request 1, then send request 3, then wait, then repeat
after job 3 is finished, send request 1, then wait, repeat
after job 1 is finished, complete the stream
I feel that in order to achieve the sending of the requests in sequence, they should be concatMaped; but in the snippet above that's not possible because of the interval that would prevent each polling stream from terminating.
Could you please advise how to modify my code to achieve what I am describing?
If I understand the problem right, I would proceed like this.
First of all I would create a poll function that returns an Observable which notifies after a round of pollings, and it emits an array of all numbers for which the call to doRequest returns 'RUNNING'. Such a function would look something like this
const poll = (numbers: number[]) => {
return from(numbers).pipe(
concatMap((n) =>
doRequest(n).pipe(
filter((resp) => resp.status === 'RUNNING'),
map((resp) => resp.value)
)
),
toArray()
);
};
Then what you need to do is to recursively iterate a call the poll function until the array emitted by the Observable returned by poll is empty.
Recursion in rxjs is obtained typically with the expand operator, and this is the operator which we are going to use also in this case, like this
poll(numbers)
.pipe(
expand((numbers) =>
numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)))
)
)
.subscribe(console.log);
A complete example can be seen in this stackblitz.
UPDATE
If the objective is to notify the job ids which have finished with a polling logic, the structure of the solution remains the same (a poll function and recursivity via expand) but the details are different.
The poll function makes sure we emit all the responses of a polling round and it looks like this:
const poll = (
numbers: number[]
) => {
console.log(`Polling ${numbers}`);
return from(numbers).pipe(
concatMap((n) => doRequest(n)),
toArray()
);
};
The recursion logic makes sure that all jobs that are still with "RUNNING" status are polled again but then we filter only the jobs which are FINISHED and passed them downstream. In other words the logic looks like this
poll(start)
.pipe(
expand((responses) => {
const numbers = responses.filter(r => r.status === 'RUNNING').map(r => r.value)
return numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)));
}),
map(responses => responses.filter(r => r.status === 'FINISHED')),
filter(finished => finished.length > 0)
)
.subscribe({
next: responses => console.log(`Job finished ${responses.map(r => r.value)}`),
complete: () => {console.log('All processed')}
});
A working example can be seen in this stackblitz.
Updated: Original answer was not on the right track.
What we want to achieve is that on each go around of the interval we poll all the outstanding jobs in order. We yield up any completed jobs to the output observable and we also omit those completed jobs from subsequent polls.
We can do that by using a Subject instead of a static observable of the job IDs. We start our poll interval and we use withLatestFrom to include the latest list of job IDs. We can then add a tap into the output observable when we get a finished job and update the Subject to omit that job.
To end the poller interval we can create an observable that fires when the array of outstanding jobs is empty and use takeUntil with that.
const number$ = new Subject();
const noMoreNumber$ = number$.pipe(skipWhile((numbers) => numbers.length > 0));
const printout$ = interval(5000).pipe(
withLatestFrom(number$),
switchMap(([_, numbers]) => {
return numbers.map((number) => defer(() => doRequest(number)));
}),
concatAll(),
//tap(console.log),
filter(({ status }) => status === 'FINISHED'),
withLatestFrom(number$),
tap(([{ value }, numbers]) =>
number$.next(numbers.filter((num) => num != value))
),
map(([item]) => item),
takeUntil(noMoreNumber$)
);
printout$.subscribe({
next: console.log,
error: console.error,
complete: () => console.log('COMPLETE'),
});
number$.next([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
The other tweak I would make is to use switchMap instead of mergeMap inside the poller itself. If you use that in combination with fromFetch for performing your HTTP calls then, if there is some long-running HTTP call which gets stuck, on the next poll the previous call will be cancelled before it makes the next HTTP call because switchMap disposes of the previous observable before subscribing to the new one.
Here's a working example:
https://stackblitz.com/edit/js-gxrrb3?devToolsHeight=33&file=index.js
Generates console output looking like this...
TRY this
import { delay, EMPTY, from, of, range } from 'rxjs';
import { concatMap, filter, mergeMap, tap, toArray } from 'rxjs/operators';
const number$ = from(range(1, 3));
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input }).pipe(delay(1000));
};
const poll = (jobs: object[]) => {
return from(jobs).pipe(
filter((job) => job['status'] !== 'FINISHED'),
concatMap((job) => doRequest(job['value'])),
tap((job) => {
console.log('polling with................', job);
}),
toArray(),
tap((result) => {
console.log('curent jobs................', JSON.stringify(result));
}),
mergeMap((result) =>
result.length > 0 ? poll(result) : of('All job completed!')
)
);
};
const initiateJob = number$.pipe(
mergeMap((id) => doRequest(id)),
toArray(),
tap((jobs) => {
console.log('initialJobs: ', JSON.stringify(jobs));
}),
concatMap(poll)
);
initiateJob.subscribe({
next: console.log,
error: console.log,
complete: () => console.log('COMPLETED'),
});

RxJs dynamic throttleTime duration a.k.a. get stream of scroll-start events

Hello,
first of all, thank you for reading this. 🙏
I want to handle scroll events stream and I want to react on scroll starts and ignore following burst of scroll events until stream considered inactive (time limit). So after delay I want repeat the same.
This is my solution so far:
import { fromEvent } from 'rxjs';
import { throttle, debounceTime } from 'rxjs/operators';
const stream = fromEvent(window, 'scroll');
const controllerStream = stream.pipe(debounceTime(500));
this.sub = stream
.pipe(
throttle(() => controllerStream, {
leading: true,
trailing: false,
})
)
.subscribe(() => {
// react on scroll-start events
});
Is there a better way?
I was considering operators like throttleTime, debounce, debounceTime... but I could not find the configuration matching my needs
Thank you 🙏👍
While this solution looks a bit involved, it achieves the behavior you describe, and can be encapsulated cleanly in a custom operator.
import { of, merge, NEVER } from 'rxjs';
import { share, exhaustMap, debounceTime, takeUntil } from 'rxjs/operators';
const firstAfterInactiveFor = (ms) => (source) => {
// Multicast the source since we need to subscribe to it twice.
const sharedSource = source.pipe(share());
return sharedSource.pipe(
// Ignore source until we finish the observable returned from exhaustMap's
// callback
exhaustMap((firstEvent) =>
// Create an observable that emits only the initial scroll event, but never
// completes (on its own)
merge(of(firstEvent), NEVER).pipe(
// Complete the never-ending observable once the source is dormant for
// the specified duration. Once this happens, the next source event
// will be allowed through and the process will repeat.
takeUntil(sharedSource.pipe(debounceTime(ms)))
)
)
);
};
// This achieves the desired behavior.
stream.pipe(firstAfterInactiveFor(500))
I have made a third version, encapsulating my solution into custom operator based on #backtick answer. Is there a problem with this solution? Memory leak or something? I am not sure whether inner controllerStream will destroy properly or at all.
const firstAfterInactiveFor = (ms) => (source) => {
const controllerStream = source.pipe(debounceTime(ms));
return source
.pipe(
throttle(() => controllerStream, {
leading: true,
trailing: false
})
)
};
// This achieves the desired behavior.
stream
.pipe(
firstAfterInactiveFor(500)
)
.subscribe(() => {
console.log("scroll-start");
});
Here is codepen with comparison of all three:
https://codepen.io/luckylooke/pen/zYvEoyd
EDIT:
better example with logs and unsubscribe button
https://codepen.io/luckylooke/pen/XWmqQBg

In RxJS, map not executing inner mapped observable

New to RxJS, but I'm trying to map a single element stream to another that produces an array after all internal/subsequent streams are finished/loaded. However, my inner observables don't seem to be executing. They just get returned cold.
High level, I need to execute http post to upload a list of files (in two different arrays to two different endpoints). Since they are large I emulate with a delay of 5 seconds. The requests need to be executed in parallel, but limited to concurrently executing X at a time (here 2). This all needs to be inside a pipe and the pipe should only allow the stream to continue after all posts are complete.
https://stackblitz.com/edit/rxjs-pnwa1b
import { map, mapTo, mergeMap, mergeAll, delay, tap, catchError, toArray } from 'rxjs/operators';
import { interval, merge, forkJoin, of, from, range, Observable } from 'rxjs';
const single = "name";
const first = ["abc", "def"];
const second = of("ghi", "jkl", "mno");
of(single)
.pipe(tap(val => console.log(`emit:${val}`)))
.pipe(
map(claim =>
merge(
from(first).pipe(map(photo => of(photo).pipe(delay(5000)))),
from(second).pipe(map(video => of(video).pipe(delay(5000))))
)
.pipe(
mergeAll(2)
)
.pipe(tap(val => console.log(`emit:${val}`)))
.pipe(toArray())
.pipe(tap(val => console.log(`emit:${val}`)))
)
)
.pipe(
catchError(error => {
console.log("error");
return Observable.throw(error);
})
)
.subscribe(val => console.log(`final:${val}`));
An inner subscribe would not wait until they are complete. Using forkJoin would not allow me to limit the concurrent uploads. How can I accomplish this?
Update:
Answer by #dmcgrandle was very helpful and led me to make the changes below that seem to be working:
import { map, mapTo, mergeMap, mergeAll, delay, tap, catchError, toArray } from 'rxjs/operators';
import { interval, merge, forkJoin, of, from, range, Observable, throwError } from 'rxjs';
const single = "name";
const first = ["abc", "def"];
const second = of("ghi", "jkl", "mno");
of(single)
.pipe(tap(val => console.log(`emit:${val}`)))
.pipe(
mergeMap(claim =>
merge(
from(first).pipe(map(photo => of(photo).pipe(delay(5000)).pipe(tap(val => console.log(`emit:${val}`))))),
from(second).pipe(map(video => of(video).pipe(delay(5000)).pipe(tap(val => console.log(`emit:${val}`)))))
)
),
mergeAll(2),
toArray()
)
.pipe(
catchError(error => {
console.log("error");
return throwError(error);
})
)
.subscribe(val => console.log(`final:${val}`));
If I am understanding you correctly, then I think this is a solution. Your issue was with the first map, which won't perform an inner subscribe, but rather just transform the stream into Observables of Observables, which didn't seem to be what you wanted. Instead I used mergeMap there.
Inside the from's I used concatMap to force each emission from first and second to happen in order and wait for one to complete before another started. I also set up postToEndpoint functions that return Observables to be closer to what your actual code will probably look like.
StackBlitz Demo
code:
import { mergeMap, concatMap, delay, tap, catchError, toArray } from 'rxjs/operators';
import { merge, of, from, concat, throwError } from 'rxjs';
const single = "name";
const first = ["abc", "def"];
const second = of("ghi", "jkl", "mno");
const postToEndpoint1$ = photo => of(photo).pipe(
tap(data => console.log('start of postTo1 for photo:', photo)),
delay(5000),
tap(data => console.log('end of postTo1 for photo:', photo))
);
const postToEndpoint2$ = video => of(video).pipe(
tap(data => console.log('start of postTo2 for video:', video)),
delay(5000),
tap(data => console.log('end of postTo2 for video:', video))
);
of(single).pipe(
tap(val => console.log(`initial emit:${val}`)),
mergeMap(claim =>
merge(
from(first).pipe(concatMap(postToEndpoint1$)),
from(second).pipe(concatMap(postToEndpoint2$))
)
),
toArray(),
catchError(error => {
console.log("error");
return throwError(error);
})
).subscribe(val => console.log(`final:`, val));
I hope this helps.

RxJS: forkJoin mergeMap

I'm trying to make multiple http requests and get returned data in one object.
const pagesToFetch = [2,3]
const request$ = forkJoin(
from(pagesToFetch)
.pipe(
mergeMap(page => this.mockRemoteData(page)),
)
)
mockRemoteData() return a simple Promise.
After first Observable emits (the once created from first entry of pagesToFetch the request$ is completed, second value in not included. How can I fix this?
You can turn each value in pagesToFetch into an Observable and then wait until all of them complete:
const observables = pagesToFetch.map(page => this.mockRemoteData(page));
forkJoin(observables)
.subscribe(...);
Or in case it's not that simple and you need pagesToFetch to be an Observable to collect urls first you could use for example this:
from(pagesToFetch)
.pipe(
toArray(),
mergeMap(pages => {
const observables = pages.map(page => this.mockRemoteData(page));
return forkJoin(observables);
}),
)
.subscribe(...);
Try the below sample format...
Observable.forkJoin(
URL 1,
URL 2
).subscribe((responses) => {
console.log(responses[0]);
console.log(responses[1]);
},
error => {console.log(error)}
);

Resources