I am batch loading data into a database and using RxJS to control the pipeline.
let event = 0;
parseStreamOfIndividualObjectsToInsert().pipe(
bufferCount(100), // batch the inserts in groups of 100
flatMap(batch => batchInsert(batch)
// I want to go on only when all batch inserts are complete
// performFurtherOperationOnlyOnce()
)
.subscribe(() => console.log('Hopefully will only log one event: ++event));
At the moment, the next method of the observer is firing after each batchInsert(), but I just want it to fire once, when all of the batch inserts have completed.
I have tried numerous combinations of concat and flatMap etc, but I still get multiple events emitted from the bulkInserts() flatMap line. Ideally, an event would only be emitted on completion.
Is it possible somehow only emit on completion?
Thanks
The toArray operation will do what you want. It'll reduce the entire stream into one event containing an array of everything:
const { bufferCount, flatMap, toArray } = require('rxjs/operators');
let event = 0;
parseStreamOfIndividualObjectsToInsert().pipe(
bufferCount(100), // batch the inserts in groups of 100
flatMap(batch => batchInsert(batch)
toArray()
)
.subscribe(() => console.log('Hopefully will only log one event: ++event));
You can use toArray to reduce the stream of observables into a list.
You need to be sure that parseStreamOfIndividualObjectsToInsert() will eventually complete. I replaced bufferCount with take in order to force the stream to complete.
let event = 0;
parseStreamOfIndividualObjectsToInsert().pipe(
take(100), // Force the stream to complete after 100 emissions
// bufferCount(100),
flatMap(batch => batchInsert(batch),
toArray() # wait for Observable to complete
)
.subscribe(() => console.log('Hopefully will only log one event: ++event));
Related
Lets consider the following example:
// Subject sources have been created from Subjects
const one$ = scheduled([firstSubjectSource$, secondSubjectSource$], asyncScheduler)
.pipe(
mergeAll(),
share(),
);
const two$ = scheduled([thirdSubjectSource$, fourthSubjectSource$], asyncScheduler)
.pipe(
mergeAll(),
share(),
);
const final$ = scheduled([one$, two$], asyncScheduler)
.pipe(
combineLatestAll(),
map(() => { /* Some mapping */ }),
);
return final$;
The final$ is created, returned and can be subscribed to.
I have observed that the marble tests work perfectly, i.e, by the time the tests run all the observables have been setup and subscribed to correctly. But in the actual executing environment (iOS 15 JavascriptCore), this doesn't seem to be the case. Values are forwarded to the ...SubjectSource$ observables after subscription to final$, but final$ never emits anything. Tapping console logs in the one$, two$ shows that they also don't emit anything. My current hypothesis is that the internal subscription process hasn't finished. I have combed through some rxjs code but it doesn't look like the subscription process is async.
AFAIK, the asyncScheduler shouldn't make the internal subscription async. It should only affect how the input values are processed and forwarded.
If the return statement is changed to below, then everything works fine. However, putting an arbitrary wait time doesn't seem like the correct thing to do either.
setTimeout(() => cb(final$), 100);
Is the internal setup of observables one$, two$ and final$ async or sync?
Is there an event that I 'need to'/'can' wait on before returning final$ for use?
How do I make sure that the observables are actually ready for use before I return it?
I have a source$ observable collecting a stream of data if there are some events trigger. I want to collect these data which occurred in a specified time into array.
const eventSubject = new Subject();
eventSubject.next(data);
const source$ = eventSubject.asObservable();
source$.pipe(takeUntil(destroyed$)).subscribe(
data => {
console.log(data);
}
);
The above source$ handle emitted data immediately.
Now i want to improve this that wait for a few seconds and collect all data happened in that specified time and emit once. So i modify to use with bufferTime like below:
const source$ = eventSubject.asObservable();
source$.pipe(takeUntil(destroyed$), bufferTime(2000)).subscribe(
data => {
console.log(data);
}
);
After testing with bufferTime, I found that it emits every 2s even source is not receiving data. If source not receiving data, it emit empty object.
What i want is only when source$ receiving data, then start to buffer for 2s, then emit value. If source$ not receiving data, it shouldn't emit anything.
I checked the bufferWhen, windowWhen, windowTime not all meeting my requirements. They are emitting every time interval specified.
Is there have other operator can do what i want?
Thanks a lot.
You can just add a filter operator to ignore the empty object emission
const source$ = eventSubject.asObservable();
source$.pipe(takeUntil(destroyed$), bufferTime(2000),filter(arr=>arr.length)).subscribe(
data => {
console.log(data);
}
);
I'd go for connect(shared$ => ...) and buffer(signal$).
I think something along these lines:
source$.pipe(
connect(shared$ => shared$.pipe(
buffer(shared$.pipe(
debounceTime(2000)
))
))
)
connect creates a shared observable so that you can have multiple subscriptions on the source without actually opening those subscriptions to it.
In there I run a buffer, whose selector is the debounceTime of the same source, so that it debounces for that much (i.e. will emit the array when source$ doesn't emit for more than 2 seconds)
Maybe what you need is throttleTime(2000, { leading: false, trailing: true }) instead of debounceTime. It depends on your use case.
The optimal solution for this case is to use buffer operator with a notifier of debouceTime operator.
For example :
const source$ = eventSubject.asObservable();
source$
.pipe(
// buffer: accumulate emitions to an array,until closing notifier emits. (closing notifier is the argument below)
buffer(
// debounceTime : will grab the emit that afterwards 2 seconds has passed without another emit.
$source.pipe(debounceTime(2000))
).subscribe(
// will return the data that has been emitted througout the 2 seconds in a form of an array , where each item in the array is the emits, by the order they were triggered.
data => {
console.log(data);
}
);
buffer:
Buffers the source Observable values until closingNotifier emits.
debounceTime:
Emits a notification from the source Observable only after a particular time span has passed without another source emission.
This solution in contrast of the filter operator solution, will not keep an interval/timer alive.
And its pretty clean and elegant IMO.
I'm creating my source observable like this (make api call every 5s):
const obs$ = Observable.interval(5000).switchMap(() => makeApiCall());
And I want to modify $obs so that it has the following characteristics:
start the observable only when at there's at least 1 subscriber
multicast. I.e. if I obs$.subscribe(...) twice, the underlying code makeApiCall() should only run once.
any subscriber which subscribes at any time should have immediately the last emitted value (and not wait ~5s until the next value emits)
retryable. If one makeApiCall() errors, I want (if possible) all subscribers to get an error notification, but reconnect to $obs, and continue doing makeApiCall() every 5s
So far I found the following leads:
It seems like I'd need to create a BehaviorSubject myBehaviorSubject, do a single subscription obs$.subscribe(myBehaviorSubject), and any other observers should subscribe to myBehaviorSubject. Not sure if that answers the "retryable" part.
I also looked at shareReplay, seems like $obs.shareReplay(1) would do the trick (for the 4 requirements). If I understood correctly it subscribes a ReplaySubject(1) to the source observable, and future observers subscribe to this ReplaySubject. Is there an equivalent shareBehavior?
In RxSwift, I found shareReplayLatestWhileConnected, which seems like the shareBehavior I was imagining. But it doesn't exist in RxJS.
Any ideas what is the best way to achieve this?
As you mentioned, shareReplay(1) pretty much gets you there. It will multicast the response to current subscribers and replay the last value (if there is one) to new subscribers. That seems like what you would want rather than shareBehavior (if it existed) since you are calling an api and there isn't an initial value.
You should know that shareReplay will create a subscription to the source stream but will only unsubscribe when refCount === 0 AND the source stream terminates (error or complete). This means that after the first subscription that the interval will start and even when there are no more subscriptions it will continue.
If you want to stop the interval when no-one is subscribed then use multicast(new ReplaySubject(1)).refCount(). The multicast operator will create a single subscription to the source stream and push all values into the subject provided as an instance (multicast(new Subject())) or by the factory (multicast(() => new Subject())). All subscribers to the stream after the multicast will subscribe to the multicast subject. So when a value flows through the multicast operator all of its subscribers will get that value. You can change the type of subject that you pass to multicast to change its behavior. In your case you probably want a ReplaySubject so that it will replay the last value to a new subscriber. You could use a BehaviorSubject too if you felt that met your need.
Now the multicast operator is connectable meaning that you would have to call connect() on the stream to make it hot. The refCount operator basically makes a connectable observable act like an ordinary observable in that it will become hot when subscribed but will become cold when there are no subscribers. It does this be keeping an internal reference count (hence the name refCount). When refCount === 0 it will disconnect.
This is the same thing as shareReplay(1) with one minor but important difference which is that when there are no more subscribers that it will unsubscribe from the source stream. If you are using a factory method to create a new subject when subscribing to the source (ex: multicast(() => new ReplaySubject(1))) then you will lose your value when the stream goes from hot to cold to hot since it will create a new subject each time it goes hot. If you want to keep the same subject between source subscriptions then you can pass in a subject instead of a factory (ex: multicast(new ReplaySubject(1)) or use its alias publishReplay(1).
As far as your last requirement of providing errors to your subscribers and then resubscribing, you can't call the error callback on a subscription and then continue getting values on the next callback. An unhandled error will end a subscription if it reaches it. So you have to catch it before it gets there and turn it into a normal message if you want your subscription to see it and still live. You can do this like so: catch((err) => of(err)) and just flag it somehow. If you want to mute it then return empty().
If you want to retry immediately then you could use the retryWhen operator but you probably want to put that before the sharing operator to make it universal. However this also prevents your subscribers from knowing about an error. Since the root of your stream is an interval and the error came from the inner observable returned from the switchMap, the error will not kill the source of the stream but it could kill the subscription. So as long as you handle the error (catch/catchError) the api call will be retried on the next interval.
Also, you may want timer(0, 5000) instead of interval so that your api call immediately fires and then fires on a 5 second interval after that.
So I would suggest something like the following:
let count = 0;
function makeApiCall() {
return Rx.Observable.of(count++).delay(1000);
}
const obs$ = Rx.Observable.timer(0, 5000)
.switchMap(() => makeApiCall().catch(() => Rx.Observable.empty()))
.publishReplay(1)
.refCount();
console.log('1 subscribe');
let firstSub = obs$.subscribe((x) => { console.log('1', x); });
let secondSub;
let thirdSub;
setTimeout(() => {
console.log('2 subscribe');
secondSub = obs$.subscribe((x) => { console.log('2', x); });
}, 7500);
setTimeout(() => {
console.log('1 unsubscribe');
firstSub.unsubscribe();
console.log('2 unsubscribe');
secondSub.unsubscribe();
}, 12000);
setTimeout(() => {
console.log('3 subscribe');
thirdSub = obs$.subscribe((x) => { console.log('3', x); });
}, 17000);
setTimeout(() => {
console.log('3 unsubscribe');
thirdSub.unsubscribe();
}, 30000);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.10/Rx.min.js"></script>
For convenience, here are aliases for multicast:
publish() === multicast(new Subject())
publishReplay(#) === multicast(new ReplaySubject(#))
publishBehavior(value) === multicast(new BehaviorSubject(value))
I just tried to implement this with rxjs 6, but the implementation feels kinda hacky. I think there should be a much cleaner way to achieve this.
The expected behavior is:
As long as there are observers, they all get the same values.
When there are 0 observers, the source subscription is closed but the ReplaySubject is not completed.
When new observers subscribe again they get the last N values and a new subscription to source is established.
When the source completes or throws an error, current observers are completed resp. notified.
After source completion or source error, new subscribers don't get replayed values any more and are completed immediately.
export function shareReplayLatestWhileConnected<T>(count?: number) {
return function (source: Observable<T>): Observable<T> {
let done = false;
return source.pipe(
// Identify when source is completed or throws an error.
tap(
null,
() => (done = true),
() => (done = true),
),
multicast(
// Subject for multicasting
new ReplaySubject<T>(count),
// Selector function. Stop subscription on subject, when source is done, to kill all subscriptions.
(shared) => shared.pipe(takeWhile(() => !done)),
),
// I was not able to get rid of duplicate subscriptions. Multicast subscribed multiple times on the source.
share(),
);
};
}
Any tips on how I could improve this solution are very appreciated.
Use it like this:
const shared$ = source$.pipe(shareReplayLatestWhileConnected(1));
I am working on my first Angular 2 product. I created this Observable.timer and then subscribed to it. It polls the server for data.
How do I pause and unpause this?
ngOnInit(): void {
this.poll = Observable.timer(500,500).subscribe((t: any) => {
this.request.getRackList(+this.global.zoneId).subscribe(
incoming => this.import(incoming),
err => { this.global.err(err) }
);
})
this.canOpenClose = (this.global.mode === 2);
}
There are multiple approaches to pause an Observable. You could:
Unsubscribe / resubscribe when needed
.filter() out emissions when paused
create a connectable Observable so you can pause / start emissions to all subscribers
You can have a look here : RxJS - pause, upon resume give last paused value
Basically, you use the switchMap operator to stop the timer emissions/restart them and a pauser subject to emit a boolean indicating whether you want to pause or restart.
If you don't need the latest value, you can replace publishReplay by publish
I want to poll for changes and when a desired value is reached the Observable should complete (or wait until timeout). Right now I use the filter which works fine to wait until the desired value is reached. But I want the Observable to push events while waiting for this value.
For example, I wait for the status 'success' and until the status changes to 'success' the status 'testing' is returned from my service. But since the filter is waiting for 'success', 'testing' never returns.
My code right now:
return Observable
.interval(this.POLL_TIMEOUT)
.flatMap(() => this.getSingleProjectStatus(projectId, repoName))
.filter(data => this.finishedStatus(data.status))
.take(1)
.timeout(this.MAX_TIMEOUT, Observable.throw(new Error('Timeout')));
You probably want takeWhile instead of filter.
return Observable
.interval(this.POLL_TIMEOUT)
.flatMap(() => this.getSingleProjectStatus(projectId, repoName))
.takeWhile(data => this.finishedStatus(data.status))
.timeout(this.MAX_TIMEOUT, Observable.throw(new Error('Timeout'));
Note the above takes all except the last event, if you want the last event too you'll need to be a little trickier.
const source = Observable.interval(this.POLL_TIMEOUT)
.flatMap(() => this.getSingleProjectStatus(projectId, repoName))
.share();
source
.takeUntil(source.filter(data => this.finishedStatus(data.status)))
.timeout(this.MAX_TIMEOUT, Observable.throw(new Error('Timeout'));
In this case you are taking all the results until another Observable emits, the other Observable in this case is simply the source filtered such that it only emits success events.
JsBin: http://jsbin.com/sojosuhune/edit?html,js,console,output