I have a data stream, with rapidly incoming data. I want to insert them into a database by keeping order. I have a database, which returns a promise, which is resolved when an insert is successful.
I would like to make an Rx stream, which buffers the new data, until the buffered data is inserted.
How can I do that?
I believe to get exactly what you desire you would need to create your own operator. Breaking from RxJS slightly you can get something like (warning, have not tested)...
export class BusyBuffer<T> {
private itemQueue = new Subject<T>();
private bufferTrigger = new Subject<{}>();
private busy = false;
constructor(consumerCallback: (items: T[]) => Promise<void>) {
this.itemQueue.buffer(this.bufferTrigger).subscribe(items => {
this.busy = true;
consumerCallback(items).then(() => {
this.busy = false;
this.bufferTrigger.next(null);
});
});
}
submitItem(item: T) {
this.itemQueue.next(item);
if(!busy) {
this.bufferTrigger.next(null);
}
}
}
Which can then be used as
let busyBuffer = new BusyBuffer<T>(items => {
return database.insertRecords(items);
});
items.subscribe(item => busyBuffer.submitItem(item));
It isn't exactly purely reactive though and someone may be able to come up with something better.
Related
I asked a question
Is Observable from chained promises equivalent of observables created with from and chained with concatMap?
on totally false premises. It seems that neither of my solutions had nothing to do with my intention.
I created a method that returns Observable and calls 2 methods returning Promise. I tried 2 ways:
public setItemInfos(itemInfos: IItemInfo[]): Observable<number> {
return from(this.db.selectionItemInfos.clear().then(() => {
return this.db.selectionItemInfos.bulkAdd(itemInfos);
}));
}
public setItemInfos(itemInfos: IItemInfo[]): Observable<number> {
const clear$ = from(this.db.selectionItemInfos.clear());
const bulkAdd$ = from(this.db.selectionItemInfos.bulkAdd(itemInfos));
return clear$.pipe(concatMap(() => bulkAdd$))
}
the use will be:
myService.setItemInfos(itemInfos).subsribe(count => {
console.log(`Cleared the table 1st and then added ${count} new items`);
});
I thought from both versions that:
table clear is execution is finished when bulkAdd starts
when bulkAdd is finished i get the count from that in subscribe
How this should really be done? Or can it be done?
This is (from what I can tell here), how I would do it.
In general, defer (or any higher-order operator) is a better way to create an observable from a promise. Defer lets you take the eager evaluation semantics of promises and turn them into the lazy evaluation semantics of observables.
Then all the usual observable operators and such will function as expected.
public setItemInfos(itemInfos: IItemInfo[]): Observable<number> {
const clear$ = defer(() => this.db.selectionItemInfos.clear());
const bulkAdd$ = defer(() => this.db.selectionItemInfos.bulkAdd(itemInfos));
return concat(clear$, bulkAdd$);
}
Update 1:
So I think I might know what you're after. This isn't really idiomatic RxJS since it's such an interleaving mix of declarative, imperative style of code. Even so, this should work? I haven't tested it fully, but some tinkering and I think this should do what you're after.
There's most assuredly a better way to accomplish the same thing, but without seeing the bigger picture of what you're after, it's hard to say.
interface Tagged<T> {
payload: T,
tag: number
}
class abitraryClass{
private setItemInfoSub: Subject<Tagged<IItemInfo[]>>;
private processItemInfo: Observable<Tagged<number>>;
private itemInfoTag = 0;
constructor(){
this.setItemInfoSub = new Subject<Tagged<IItemInfo[]>>();
this.processItemInfo = this.setItemInfoSub.pipe(
concatMap(({tag, payload: itemInfos}) => this.db.selectionItemInfos.clear().pipe(
ignoreElements(),
concatWith(defer(() => this.db.selectionItemInfos.bulkAdd(itemInfos))),
map(response => ({
payload: response,
tag
}))
)),
shareReplay(1)
);
// Make the processing pipeline live at all times.
this.processItemInfo.subscribe();
}
public setItemInfos(itemInfos: IItemInfo[]): Observable<number> {
const myTag = this.itemInfoTag++;
this.setItemInfoSub.next({
payload: itemInfos,
tag: myTag
});
return this.processItemInfo.pipe(
filter(({tag}) => tag == myTag),
map(({payload}) => payload)
);
}
}
I use a derived store in the code below. It feels like a strange construct because I only use the derived construct for the dynamic $session dependency and to get the normData. But not with $norm. I use $norm only once to kick off the derived store.
Nevertheless it seem to work fine. But I have to renew the subscription if the $session changes. Is it possible to update the RxFire / RxJs subscription without unsubscribing first?
let normDocRef = null;
let normData = null;
let normSubscription = null;
const norm = derived(
session,
$session => {
normDocRef = db.doc(`uploads/${$session.a_id}_${$session.year}`);
// renew the subscription if $session changes
if (normSubscription)
normSubscription.unsubscribe();
normSubscription = doc(normDocRef).subscribe(snapshot => {
if (snapshot.exists) {
normData = snapshot.data();
} else {
normData = null;
};
});
},
);
$norm; // kick off the derived store to monitor $session
// show the data and updates
$: console.log(normData);
onDestroy(() => {
if (normSubscription) normSubscription.unsubscribe();
});
Update: I can use the set and return options of the derived store to change $norm in a real $norm Svelte store. Code below in my own answer.
But the real question is: Can I update a subscription. Change the subscription without the unsubscribe?
I already had the answer, but did not realize it.
Below the derived store code with the set() and return() options.
When the session changes the return() will unsubscribe automatically.
So still an unsubscribe and not an update ... but this feels good. Nice!
let normDocRef = null;
let normSubscription = null
const norm = derived(
session,
($session, set) => {
normDocRef = db.doc(`uploads/${$session.a_id}_${$session.year}`);
normSubscription = doc(normDocRef).subscribe(snapshot => {
if (snapshot.exists) {
set(snapshot.data());
} else {
set({}); // clear
};
});
return () => {
normSubscription.unsubscribe();
};
}, {} // initial value
);
$: console.log('$norm', $norm); // Now it is a real store
onDestroy(() => {
if (!normSubscription.closed) {
normSubscription.unsubscribe();
}
});
API docs derived store:
Derives a store from one or more other stores. Whenever those dependencies change (like the $session), the callback runs.
If you "return a function" from the callback, it will be called (before the callback) when a) the callback runs again (because the dependency changed), or b) ...
Ok, roughly get what you trying to describe over here.
You can actually use the reactive declaration to execute code when a variable / store changed.
In this case is to execute the resubscribe method:
let normDocRef = null;
let normData = null;
let normSubscription = null;
$: {
normDocRef = db.doc(`uploads/${$session.a_id}_${$session.year}`);
// renew the subscription if $session changes
if (normSubscription) {
normSubscription.unsubscribe();
normSubscription = doc(normDocRef).subscribe(snapshot => {
if (snapshot.exists) {
normData = snapshot.data();
} else {
normData = null;
};
});
}
}
onDestroy(() => {
if (normSubscription) normSubscription.unsubscribe();
});
The key here, is that when compiling this, Svelte knows that the block is depending on $session, so it will re-execute the code block whenever $session changed.
Should you want to refactor it out into another function, you need to make sure that Svelte knows that function depends on $session, ie:
$: resubscribe_norm($session);
Here, Svelte can tell that, if $session changed, need to call resubscribe_norm again.
Hi i have a global service for several applications and i wish to make a method with several subscribes in order to stock and initialize all my datas and i wish make a subscribe to this method in my appComponent but i don't know how to make that
In my service
private initData(isLogged: boolean) {
this.http.get('/api/conf').subscribe(
conf => {
this.http.get('api/param').subscribe(
tokResp => {
this.appParams.token = tkResp.queoval;
this.appParams.culture = tkResp.culture;
this.appParams.GMT = tkResp.gmt;
this.http.get('/api/trad').subscribe(
trad => {
this.label = trad
// return an Observable
}
)
}
)
}
)
}
In my AppComponent
this.service.initData().subscribe(
result => {
this.test = result
}
How can i make that? I can't find the information in the documentation. Thank you for your help. It's important for my work, i used so much time to research for nothing :(
So since you want to make multiple async requests one after the other, you should use the observable function ".flatMap" (this is very similar to a Promises ".then"). The ".flatMap" function allows you to wait until the first request is completed before you continue.
So in your case, you would want your service to look something like this:
private initData(isLogged: boolean) {
return this.http.get('/api/conf').flatMap(
conf => {
return this.http.get('api/param');
}
).flatMap(
tokResp => {
this.appParams.token = tkResp.queoval;
this.appParams.culture = tkResp.culture;
this.appParams.GMT = tkResp.gmt;
return this.http.get('/api/trad');
}
).flatMap(
trad => {
this.label = trad;
return trad;
}
);
}
This function has all of the async requests chained together through ".flatMap" so that they are only called after the previous request completes.
The component file looks fine and should work with this new service.
As a general note, you should never subscribe to the observable inside
of the service. You should instead use functions like map, flatMap,
forkJoin ...
I have an observable containing all data. I want to create a new observable which will return the filteredData.
So the filtering part must run if the data or global filter changes.
then I started like:
function setFilter(filter) {
this.filter = filter;
}
this.filteredData = this.data.pipe(map(todos => {
// the filtering inside here will only run if data changed.
// and not if "this.filter" changes..
})
);
But the problem was it did just returned the filtererData if the data changed and not if the filter changed.
So I found the following solution:
function setFilter(filter) {
this.filter = filter;
this._filterSub.next(1);
}
this.filteredData = combineLatest(this.data, filterObs, (data, filter) => {
// this works but I've got the feeling that there is a nicer way
}
Now this works but just because I made another unnecessary BehaviorSubject and abuse it to trigger the "filtering part".
Isn't there a better cleaner way?
The way to do it is to user ReplaySubject. Either as your data field or as the target of the pipe:
const filtered = new ReplaySubject(1);
// ... your other code, including updating the filter
this.data.filter(<filter>).subscribe(filtered);
I need to create an observable, which I can "pull" data from, to work with a pageable api. I can only fetch 100 items per request, I want to be able to use observable as a generator function (on which I can call .next() to issue a request to get next 100 items.
I can't unfortunately find a way to do it with Rx. I suppose it's possible using controlled observable or a subject. Can you guys show me an example.
this is what I've gotten so far:
function list(entityType, viewName, fetchAll = false) {
var skip = 0,
total = 0;
const subject = new Rx.Subject(),
response$ = subject
.takeWhile(() => skip <= total)
.startWith(skip)
.flatMap((skip) => fetchPagePromise(skip)),
next = () => subject.onNext(skip);
if (fetchAll) {
Rx.Observable.timer(100, 100).subscribe(() => next());
}
return {
data$: response$.map(response => response),
next: fetchAll === true ? undefined : next
};
function fetchPagePromise() {
let limit = 100,
obj = {
viewName, limit, skip
},
qs = objectToQueryString(obj);
return $http.get(`${apiBase}/api/data/${entityType}${qs}`).then((res) => {
total = res.data.Total;
skip += limit;
return res.data.Rows;
});
}
}
this kinda works like a generator. it returns an Observable and next handler. Whenever next is called it pulls next 100 items from api and pushes into the Observable. Also if there’s a third parameter fetchAll passed, then it will keep fetching data until there’s no more. What scares me though that there are 2 mutating vars in function's closure - skip and total, and I don't know if managing them like this in asynchronous/unpredictable environment is ok.
One of the things you generally want to avoid is trying to make Rx into a plain old event emitter. Usually it is an indicator when you try and just trigger Observables manually by passing around a Subjects observer interface.
You should ask yourself, where is my data coming from? What calls next(), what calls that, etc. After enough of these you will generally find that this will lead you to something that can be wrapped by an Observable directly rather than explicitly calling next(). Also, I think the fetchAll flag should really be kept externally. You are only making the interface confusing by essentially turning it into a void method just by passing in a flag.
So I would recommend refactoring like so:
Rx.Observable.prototype.lazyRequest = function(entityType, viewName, limit = 100) {
var source = this;
return Rx.Observable.create(obs => {
var response = source
//Skip is really just the (limit * index)
.map((x, i) => i * limit)
.flatMap((skip) => {
let obj = {viewName, skip, limit},
qs = objectToQueryString(obj);
//Handle promises implicitly
return $http.get(`${apiBase}/api/data/${entityType}${qs}`);
},
//Return this with our skip information
(skip, res) => {skip, res})
//Publish it so the stream get shared.
.publish();
//This will emit once once you are out of data
var stop = response.first(x => x.skip >= x.res.data.Total);
return new CompositeDisposable(
//Complete this stream when stop emits
response.takeUntil(stop)
//Downstream only cares about the data rows
.map(x => x.res.data.Rows)
.subscribe(obs),
//Hook everything up
response.connect());
});
}
Then you can use it like so:
//An example of a "starting point", a button click
//Update the rows every time a new event comes through
Rx.Observable.fromEvent($button, 'click')
.startWith(0) //Inject some data into the pipeline
.lazyRequest(entityType, viewName)
.subscribe(/*Do something with the returned rows*/);
//Get all of the rows, will keep hitting the endpoint until it completes
Rx.Observable.interval(100)
.lazyRequest(entityType, viewName)
//Gather all the values into an array and emit that.
.toArray()
.subscribe();