I have a list of observables I'd like to fire 5 at a time. I've tried using a mergeMap, but clearly I'm using it wrong:
// obsArray defined above... is a Array of Observables... about 30 of them
of(obsArray).pipe(
mergeMap(x => x, 5)
).subscribe();
The issue is that x in the mergeMap is the entire observable list. How do I send 5 at a time to be fired (they are http calls)?
Use from to emit single items from an array. You can also replace mergeMap(x => x) with mergeAll.
from(obsArray).pipe(mergeAll(5))
Related
I'm trying to achieve something very similar to a buffer count. As values come through the pipe, bufferCount of course buffers them and sends them down in batches. I'd like something similar to this that will emit all remaining items if there are currently fewer than the buffer size in the stream.
It's a little confusing to word, so I'll provide an example with what I'm trying to achieve.
I have something adding items individually to a subject. Sometimes it'll add 1 item a minute, sometimes it'll add 1000 items in 1 second. I wish to do a long running process (2 seconds~) on batches of these items as to not overload the server.
So for example, consider the timeline where P is processing
---A-----------B----------C---D--EFGHI------------------
|_( P(A) ) |_(P(B)) |_( P(C) ) |_(P([D, E, F, G, H, I]))
This way I can process the events in small or large batches depending on how many events are coming through, but i ensure the batches remain smaller than X.
I basically need to map all the individual emits into emits that contain chunks of 5 or fewer. As I pipe the events into a concatMap, events will start to stack up. I want to pick these stacked up events off in batches. How can I achieve this?
Here's a stackblitz with what I've got so far: https://stackblitz.com/edit/rxjs-iqwcbh?file=index.ts
Note how item 4 and 5 don't process until more come in and fill in the buffer. Ideally after 1,2,3 are processed, it'll pick off 4,5 the queue. Then when 6,7,8 come in, it'll process those.
EDIT: today I learned that bufferTime has a maxBufferSize parameter, that will emit when the buffer reaches that size. Therefore, the original answer below isn't necessary, we can simply do this:
const stream$ = subject$.pipe(
bufferTime(2000, null, 3), // <-- buffer emits # 2000ms OR when 3 items collected
filter(arr => !!arr.length)
);
StackBlitz
ORIGINAL:
It sounds like you want a combination of bufferCount and bufferTime. In other words: "release the buffer when it reaches size X or after Y time has passed".
We can use the race operator, along with those other two to create an observable that emits when the buffer reaches the desired size OR after the duration has passed. We'll also need a little help from take and repeat:
const chunk$ = subject$.pipe(bufferCount(3));
const partial$ = subject$.pipe(
bufferTime(2000),
filter(arr => !!arr.length) // don't emit empty array
);
const stream$ = race([chunk$, partial$]).pipe(
take(1),
repeat()
);
Here we define stream$ to be the first to emit between chunk$ and partial$. However, race will only use the first source that emits, so we use take(1) and repeat to sort of "reset the race".
Then you can do your work with concatMap like this:
stream$.pipe(
concatMap(chunk => this.doWorkWithChunk(chunk))
);
Here's a working StackBlitz demo.
You may want to roll it into a custom operator, so you can simply do something like this:
const stream$ = subject$.pipe(
bufferCountTime(5, 2000)
);
The definition of bufferCountTime() could look like this:
function bufferCountTime<T>(count: number, time: number) {
return (source$: Observable<T>) => {
const chunk$ = source$.pipe(bufferCount(count));
const partial$ = source$.pipe(
bufferTime(time),
filter((arr: T[]) => !!arr.length)
);
return race([chunk$, partial$]).pipe(
take(1),
repeat()
);
}
}
Another StackBlitz sample.
Since I noticed the use of forkJoin in your sample code, I can see you are sending a request to the server for each emission (I was originally under the impression that you were making only 1 call per batch with combined data).
In the case of sending one request per item the solution is much simpler!
There is no need to batch the emissions, you can simply use mergeMap and specify its concurrency parameter. This will limit the number of currently executing requests:
const stream$ = subject$.pipe(
mergeMap(val => doWork(val), 3), // 3 max concurrent requests
);
Here is a visual of what the output would look like when the subject rapidly emits:
Notice the work only starts for the first 3 items initially. Emissions after that are queued up and processed as the prior in flight items complete.
Here's a StackBlitz example of this behavior.
TLDR;
A StackBlitz app with the solution can be found here.
Explanation
Here would be an approach:
const bufferLen = 3;
const count$ = subject.pipe(filter((_, idx) => (idx + 1) % bufferLen === 0));
const timeout$ = subject.pipe(
filter((_, idx) => idx === 0),
switchMapTo(timer(0))
);
subject
.pipe(
buffer(
merge(count$, timeout$).pipe(
take(1),
repeat()
)
),
concatMap(buffer => forkJoin(buffer.map(doWork)))
)
.subscribe(/* console.warn */);
/* Output:
Processing 1
Processing 2
Processing 3
Processed 1
Processed 2
Processed 3
Processing 4
Processing 5
Processed 4
Processed 5
Processing 6 <- after the `setTimeout`'s timer expires
Processing 7
Processing 8
Processed 6
Processed 7
Processed 8
*/
The idea was to still use the bufferCount's behavior when items come in synchronously, but, at the same time, detect when fewer items than the chosen bufferLen are in the buffer. I thought that this detection could be done using a timer(0), because it internally schedules a macrotask, so it is ensured that items emitted synchronously will be considered first.
However, there is no operator that exactly combines the logic delineated above. But it's important to keep in mind that we certainly want a behavior similar to the one the buffer operator provides. As in, we will for sure have something like subject.pipe(buffer(...)).
Let's see how we can achieve something similar to what bufferTime does, but without using bufferTime:
const bufferLen = 3;
const count$ = subject.pipe(filter((_, idx) => (idx + 1) % bufferLen === 0));
Given the above snippet, using buffer(count$) and bufferTime(3), we should get the same behavior.
Let's move now onto the detection part:
const timeout$ = subject.pipe(
filter((_, idx) => idx === 0),
switchMapTo(timer(0))
);
What it essentially does is to start a timer after the subject has emitted its first item. This will make more sense when we have more context:
subject
.pipe(
buffer(
merge(count$, timeout$).pipe(
take(1),
repeat()
)
),
concatMap(buffer => forkJoin(buffer.map(doWork)))
)
.subscribe(/* console.warn */);
By using merge(count$, timeout$), this is what we'd be saying: when the subject emits, start adding items to the buffer and, at the same time, start the timer. The timer is started too because it is used to determine if fewer items will be in the buffer.
Let's walk through the example provided in the StackBlitz app:
from([1, 2, 3, 4, 5])
.pipe(tap(i => subject.next(i)))
.subscribe();
// Then mimic some more items coming through a while later
setTimeout(() => {
subject.next(6);
subject.next(7);
subject.next(8);
}, 10000);
When 1 is emitted, it will be added to the buffer and the timer will start. Then 2 and 3 arrive immediately, so the accumulated values will be emitted.
Because we're also using take(1) and repeat(), the process will restart. Now, when 4 is emitted, it will be added to the buffer and the timer will start again. 5 arrives immediately, but the number of the collected items until now is less than the given buffer length, meaning that until the 3rd value arrives, the timer will have time to finish. When the timer finishes, the [4,5] chunk will be emitted. What happens with [6, 7, 8] is the same as what happened with [1, 2, 3].
of([1,2,3]).subscribe(console.log)
prints:[1,2,3]
But:
of([1,2,3]).pipe(concatAll()).subscribe(console.log)
prints:
1
2
3
Why the above happens? Why adding concatAll() emits the elements of the array one by one? Isn't this somehow the opposite of what the word concat means?
I feel that concatAll() acts differently depending on the input.
Consider also this:
from([of(1),of(2),of(3)]).pipe(concatAll()).subscribe(console.log)
It will again print:
1
2
3
So of([1,2,3]).pipe(concatAll()) == from([of(1),of(2),of(3)]).pipe(concatAll())
But of([1,2,3]) != from([of(1),of(2),of(3)]) because subscribing to the latter will print:
Observable { _isScalar: false, _subscribe: [Function] }
Observable { _isScalar: false, _subscribe: [Function] }
Observable { _isScalar: false, _subscribe: [Function] }
The right side of the above equality is pretty clear to me, but where is documented that concatAll() should emit all the values of the array separately, acting like a pipeable from?
All the operators that deal with higher-order observable operators, perform the same conversion that RxJS's from operator does.
that is:
mergeMap(_ => ([1,2,3]))
Is basically the same as
mergeMap(_ => from([1,2,3]))
That works the same when merging/switching/concatenating promises and other iterables.
The difference you're seeing in behaviour is also attributed to the fact that from creates a new observable, while the higher-order observable operators tend to create a new observables and define behaviour around how those observables are subscribed to.
so concatAll is adding extra behaviour.
Finally, just because two things have similar outputs for a given example, doesn't mean they're doing the same thing under the hood.
of([1,2,3]).pipe(concatAll()) and from([of(1),of(2),of(3)]).pipe(concatAll()) describe two very different set of behaviours that in (this case) give you the same output to the console.
The first emits an array and has that array turned into a stream of numbers by concatAll(). The second emits three observables, and has has concatAll() subscribe to each only when the previous one completes. They complete after emitting just one number.
You can see the distinction clearly with your second example:
of([1,2,3]) and from([of(1),of(2),of(3)])
The first emits and array, the second emits three observables. That hasn't changed.
concat([1,2,3]), &&
concat(of(1), of(2), of(3))
Emit the same output again as well.
I have a hard time understanding something related to using a concat function inside a mergeMap operator.
The documentation for concat says that
You can pass either an array of Observables, or put them directly as arguments.
When I put the Observables directly as arguments, like in the following example, I correctly get 20 and 24 in the console.
of(4)
.pipe(
mergeMap(number => concat(of(5 * number), of(6 * number)))
)
.subscribe(value => console.log(value));
But when I put them as an array, then in the console I get the Observables and not their values:
of(4)
.pipe(
mergeMap(number => concat([of(5 * number), of(6 * number)]))
)
.subscribe(value => console.log(value));
Here's a live version in Stackblitz.
Any idea why is that? Shouldn't both examples work identically?
Those two scenarios are different and they should not work identically. concat takes Observables as arguments and it will sequentially subscribe to those streams and only subscribe to the next Observable when the previous one completed. Every operator or creation method returns an Observable. This means that in the first example, when you are using concat, it will return an Observable that emits 20 and then 24. Because you are dealing with a nested Observable you have to use mergeMap which will subscribe to the resulting Observable returned by concat.
Now in the second example, if you pass in an array, concat will convert this (using from() internally) to an Observable that emits 2 values, and those values are Observables again. So you have 3 levels of nesting here. The first is the most outer Observable, the source, which of(4), the second level is the one you map to inside your mergeMap and the third in the second example are the Observables inside your array. The thing is you only flatten the levels up to level 2 but not the 3rd level. Again, in your second example the Observable returned by mergeMap emits two Observables but those are just the proxies and not the values emitted by these Observables. If you want to subscribe to those as well, you could chain on another mergeMap like so
concatArray() {
of(4)
.pipe(
mergeMap(number => concat([of(5 * number), of(6 * number)])),
mergeMap(x => x)
)
.subscribe(value => console.log(value));
}
Another way is to spread the array so that concat does not receive an object that is ArrayLike but rather Observables directly:
concatArray() {
of(4)
.pipe(
mergeMap(number => concat(...[of(5 * number), of(6 * number)]))
)
.subscribe(value => console.log(value));
}
Both will print out:
20
24
I hope this makes this a little bit more clear and describes the differences between the first and the second example.
As title suggest, both function seems to have similar effect and return emit nothing when predicate does not match. It looks like skipWhile is the reverse of filter ?
As #cartant says
skipWhile "... emits all further source items as soon as the condition becomes false"
Note that skipWhile takes a predicate (expression that returns a boolean).
In addition there is also skipUntil, which this takes an observable and values will be skipped until that observable emits anything.
So sometimes the following can be useful (pan is a hammerjs event btw.):
// When event.deltaX reaches 20 emit true to indicate a threshold is reached
const thresholdReached$ = pan$.pipe(map(e => Math.abs(e.deltaX) > 20 ), first(v => v), shareReplay(1));
// Emit all events, but skip them until the first time that thresholdReached$ emits
const panMove$ = pan$.pipe(skipUntil(thresholdReached$));
Important to realize that the observable passed to skipUntil can emit any value to trigger the skipping to stop. It doesn't have to be a 'truthy' value.
So if you have var b = new BehaviorSubject(false) then skipUntil(b) will immediately stop skipping and you will quickly get very confused!
According to the documentation, the buffer transform will wait for a delay before emitting any values. What I'd like is to get the current value immediately, then only update every X seconds.
I've not been able to achieve this with rxjs yet. The closest I've come is to bind the observable then use a setTimeout function to rebind after the buffer timeout occurs. This has a side effect of clearing the current value for those X seconds before emitting the current values.
Any ideas?
Thanks!
Assuming by "current value immediately" you mean "first value as soon as it emits", you can buffer on the second element to the last, and merge in the first:
// source$: Observable<T>
const pub_source$ = source$.publish();
Observable.merge(
pub_source$.take(1).map(first => [first]),
pub_source$.skip(1).buffer(Observable.interval(X))
);
pub_source$.connect();
The source needs to be cold so that take(1) and skip(1) relate to the same element, so we use publish. The first element is also wrapped to keep the output type T[] consistent.
There's an easier way using the zip operator, see lightbulb note in learnrxjs
Combined with interval or timer, zip can be used to time output from another source!
// Useful for slow source that emits at around the same rate as interval
// but suffers back-pressure with fast-emitting source
const interval = 1000
const output = Observable.zip(source, Observable.timer(0, interval))
.map(x => x[0])
The Observable.timer 'regulates' the output from source. Note, timer's first parameter sets the delay for the first emit.
Working example: CodePen
Footnote
I just realized this will create back-pressure (build-up of un-emitted values) if you have a lot of events per second, so buffer is the better way to go with a fast emitting source.
// Buffered version for fast source
const output2 = source.buffer(Observable.timer(0, interval))
.filter(x => x.length) // filter out empty buffer emits
.flatMap(x => Observable.from(x)) // optional, converts array back to single emits