So I'm attempting to use reactives to recompose chunked messages identified by ID and am having a problem terminating the final observable. I have a Message class which consists of Id, Total Size, Payload, Chunk Number and Type and have the following client-side code:
I need to calculate the number of messages to Take at runtime
(from messages in
(from messageArgs in Receive select Serializer.Deserialize<Message>(new MemoryStream(Encoding.UTF8.GetBytes(messageArgs.Message))))
group messages by messages.Id into grouped select grouped)
.Subscribe(g =>
{
var cache = new List<Message>();
g.TakeWhile((int) Math.Ceiling(MaxPayload/g.First().Size) < cache.Count)
.Subscribe(cache.Add,
_ => { /* Rebuild Message Parts From Cache */ });
});
First I create a grouped observable filtering messages by their unique ID and then I am trying to cache all messages in each group until I have collected them all, then I sort them and put them together. The above seems to block on g.First().
I need a way to calculate the number to take from the first (or any) of the messages that come through however am having difficulty doing so. Any help?
First is a blocking operator (how else can it return T and not IObservable<T>?)
I think using Scan (which builds an aggregate over time) could be what you need. Using Scan, you can hide the "state" of your message re-construction in a "builder" object.
MessageBuilder.IsComplete returns true when all the size of messages it has received reaches MaxPayload (or whatever your requirements are). MessageBuilder.Build() then returns the reconstructed message.
I've also moved your "message building" code into a SelectMany, which keeps the built messages within the monad.
(Apologies for reformatting the code into extension methods, I find it difficult to read/write mixed LINQ syntax)
Receive
.Select(messageArgs => Serializer.Deserialize<Message>(
new MemoryStream(Encoding.UTF8.GetBytes(messageArgs.Message))))
.GroupBy(message => message.Id)
.SelectMany(group =>
{
// Use the builder to "add" message parts to
return group.Scan(new MessageBuilder(), (builder, messagePart) =>
{
builder.AddPart(messagePart);
return builder;
})
.SkipWhile(builder => !builder.IsComplete)
.Select(builder => builder.Build());
})
.Subscribe(OnMessageReceived);
Related
I am working on a Project where our client generates almost 500 request simultaneously. I am using the forkJoin to get all the responses as Array.
But the Server after 40-50 request Blocks the requests or sends only errors. I have to split these 500 requests in Chunks of 10 requests and loop over this chunks array and have to call forkJoin for each chunk, and convert observable to Promise.
Is there any way to get rid of this for loop over the chucks?
If I understand right you question, I think you are in a situation similar to this
const clientRequestParams = [params1, params2, ..., params500]
const requestAsObservables = clientRequestParams.map(params => {
return myRequest(params)
})
forkJoin(requestAsObservables).subscribe(
responses => {// do something with the array of responses}
)
and probably the problem is that the server can not load so many requests in parallel.
If my understanding is right and if, as you write, there is a limit of 10 for concurrent requests, you could try with mergeMap operator specifying also the concurrent parameter.
A solution could therefore be the following
const clientRequestParams = [params1, params2, ..., params500]
// use the from function from rxjs to create a stream of params
from(clientRequestParams).pipe(
mergeMap(params => {
return myRequest(params)
}, 10) // 10 here is the concurrent parameter which limits the number of
// concurrent requests on the fly to 10
).subscribe(
responseNotification => {
// do something with the response that you get from one invocation
// of the service in the server
}
)
If you adopt this strategy, you limit the concurrency but you are not guaranteed the order in the sequence of the responses. In other words, the second request can return before the first one has returned. So you need to find some mechanism to link the response to the request. One simple way would be to return not only the response from the server, but also the params which you used to invoke that specific request. In this case the code would look like this
const clientRequestParams = [params1, params2, ..., params500]
// use the from function from rxjs to create a stream of params
from(clientRequestParams).pipe(
mergeMap(params => {
return myRequest(params).pipe(
map(resp => {
return {resp, params}
})
)
}, 10)
).subscribe(
responseNotification => {
// do something with the response that you get from one invocation
// of the service in the server
}
)
With this implementation you would create a stream which notifies both the response received from the server and the params used in that specific invocation.
You can adopt also other strategies, e.g. return the response and the sequence number representing that response, or maybe others.
i thought i got the hang of dexie, but now i'm flabbergasted:
two tables, each with a handful of records. Komps & Bretts
output all Bretts
rdb.Bretts.each(brett => {
console.log(brett);
})
output all Komps
rdb.Komps.each(komp=> {
console.log(komp);
})
BUT: this only outputs the Bretts, for some weird reason, Komps is empty
rdb.Bretts.each(brett => {
console.log(brett);
rdb.Komps.each(komp=> {
console.log(komp);
})
})
i've tried all kinds of combinations with async/await, then() etc, the inner loop cannot find any data in the inner table, whatever table i want to something with.
2nd example. This Works:
await rdb.Komps.get(163);
This produces an error ("Failed to execute 'objectStore' on 'IDBTransaction…ction': The specified object store was not found.")
rdb.Bretts.each(async brett => {
await rdb.Komps.get(163);
})
Is there some kind of locking going on? something that can be disabled?
Thank you!
Calling rdb.Bretts.each() will implicitly launch a readOnly transaction limited to 'Bretts' only. This means that within the callback you can only reach that table. And that's the reason why it doesn't find the Comps table at that point. To get access to the Comps table from within the each callback, you would need to include it in an explicit transaction block:
rdb.transaction('r', 'Komps', 'Bretts', () => {
rdb.Bretts.each(brett => {
console.log(brett);
rdb.Komps.each(komp=> {
console.log(komp);
});
});
});
However, each() does not respect promises returned by the callback, so even this fix would not be something that I would recommend either - even if it would solve your problem. You could easlily get race conditions as you loose the control of the flow when launching new each() from an each callback.
I would recommend you to using toArray(), get(), bulkGet() and other methods than each() where possible. toArray() is also faster than each() as it can utilize faster IDB Api IDBObjectStore.getAll() and IDBIndex.getAll() when possible. And you don't nescessarily need to encapsulate the code in a transaction block (unless you really need that atomicy).
const komps = await rdb.Komps.toArray();
await Promise.all(
komps.map(
async komp => {
// Do some async call per komp:
const brett = await rdb.Bretts.get(163));
console.log("brett with id 163", brett);
}
)
);
Now this example is a bit silly as it does the exact same db.Bretts.get(163) for each komp it founds, but you could replace 163 with some dynamic value there.
Conclusion: There are two issues.
The implicit transaction of Dexie's operation and the callback to each() lives within that limited transaction (tied to one single table only) unless you surround the call with a bigger explicit transaction block.
Try avoid to start new async operation within the callback of Dexie's db.Table.each() as it does not expect promises to be returned from its callback. You can do it but it is better to stick with methods where you can keep control of the async flow.
I am trying to aggregate/tabulate the results of a set of observables. I have an array of observables that each return a number and I want to total up those results and emit that as the value. Each time the source numbers change, I want the end result to reflect the new total. The problem is that I am getting the previous results added to the new total. This has to do with how I am using the reduce/scan operator. I believe it needs to be nested inside a switchMap/mergeMap, but so far I have been unable to figure out the solution.
I mocked up a simple example. It shows how many cars are owned by all users in total.
Initially, the count is correct, but when you add a car to a user, the new total includes the previous total.
https://stackblitz.com/edit/rxjs-concat-observables-3-drfd36
Any help is greatly appreciated.
Your scan works perfectly right, the point is that for each update the stream gets all data repetitively, so, the fastest way to fix I think is to set a new instance of the stream at the handleClickAddCar.
https://stackblitz.com/edit/rxjs-wrong-count.
I ended up doing this:
this.carCount$ = this.users$.pipe(
map((users: User[]): Array<Observable<number>> => {
let requests = users.map(
(user: User): Observable<number> => {
return this.store.select(UserSelectors.getCarsForUser(user)).pipe(
map((cars: Car[]): number => {
return cars.length;
})
);
}
);
return requests;
}),
flatMap((results): Observable<number> => {
return combineLatest(results).pipe(
take(1),
flatMap(data => data),
reduce((accum: number, result: number): number => {
return accum + result;
}, 0)
)
})
);
I think the take(1) ends up doing the same thing as Yasser was doing above by recreating the entire stream. I think this way is a little cleaner.
I also added another stream below it (in the code) that does one level deeper in terms of retrieving observables of observables.
https://stackblitz.com/edit/rxjs-concat-observables-working-1
Anyone have a cleaner, better way of doing this type of roll-up of observable results?
I have written a bit of code that allows a user to upvote / downvote recipes in a manner similar to Reddit.
Each individual vote is stored in a Firestore collection named votes, with a structure like this:
{username,recipeId,value} (where value is either -1 or 1)
The recipes are stored in the recipes collection, with a structure somewhat like this:
{title,username,ingredients,instructions,score}
Each time a user votes on a recipe, I need to record their vote in the votes collection, and update the score on the recipe. I want to do this as an atomic operation using a transaction, so there is no chance the two values can ever become out of sync.
Following is the code I have so far. I am using Angular 6, however I couldn't find any Typescript examples showing how to handle multiple gets() in a single transaction, so I ended up adapting some Promise-based JavaScript code that I found.
The code seems to work, but there is something happening that is concerning. When I click the upvote/downvote buttons in rapid succession, some console errors occasionally appear. These read POST https://firestore.googleapis.com/v1beta1/projects/myprojectname/databases/(default)/documents:commit 400 (). When I look at the actual response from the server, I see this:
{
"error": {
"code": 400,
"message": "the stored version (1534122723779132) does not match the required base version (0)",
"status": "FAILED_PRECONDITION"
}
}
Note that the errors do not appear when I click the buttons slowly.
Should I worry about this error, or is it just a normal result of the transaction retrying? As noted in the Firestore documentation, a "function calling a transaction (transaction function) might run more than once if a concurrent edit affects a document that the transaction reads."
Note that I have tried wrapping try/catch blocks around every single operation below, and there are no errors thrown. I removed them before posting for the sake of making the code easier to follow.
Very interested in hearing any suggestions for improving my code, regardless of whether they're related to the HTTP 400 error.
async vote(username, recipeId, direction) {
let value;
if ( direction == 'up' ) {
value = 1;
}
if ( direction == 'down' ) {
value = -1;
}
// assemble vote object to be recorded in votes collection
const voteObj: Vote = { username: username, recipeId: recipeId , value: value };
// get references to both vote and recipe documents
const voteDocRef = this.afs.doc(`votes/${username}_${recipeId}`).ref;
const recipeDocRef = this.afs.doc('recipes/' + recipeId).ref;
await this.afs.firestore.runTransaction( async t => {
const voteDoc = await t.get(voteDocRef);
const recipeDoc = await t.get(recipeDocRef);
const currentRecipeScore = await recipeDoc.get('score');
if (!voteDoc.exists) {
// This is a new vote, so add it to the votes collection
// and apply its value to the recipe's score
t.set(voteDocRef, voteObj);
t.update(recipeDocRef, { score: (currentRecipeScore + value) });
} else {
const voteData = voteDoc.data();
if ( voteData.value == value ) {
// existing vote is the same as the button that was pressed, so delete
// the vote document and revert the vote from the recipe's score
t.delete(voteDocRef);
t.update(recipeDocRef, { score: (currentRecipeScore - value) });
} else {
// existing vote is the opposite of the one pressed, so update the
// vote doc, then apply it to the recipe's score by doubling it.
// For example, if the current score is 1 and the user reverses their
// +1 vote by pressing -1, we apply -2 so the score will become -1.
t.set(voteDocRef, voteObj);
t.update(recipeDocRef, { score: (currentRecipeScore + (value*2))});
}
}
return Promise.resolve(true);
});
}
According to Firebase developer Nicolas Garnier, "What you are experiencing here is how Transactions work in Firestore: one of the transactions failed to write because the data has changed in the mean time, in this case Firestore re-runs the transaction again, until it succeeds. In the case of multiple Reviews being written at the same time some of them might need to be ran again after the first transaction because the data has changed. This is expected behavior and these errors should be taken more as warnings."
In other words, this is a normal result of the transaction retrying.
I used RxJS throttleTime to prevent the user from flooding the Firestore server with transactions by clicking the upvote/downvote buttons in rapid succession, and that greatly reduced the occurrences of this 400 error. In my app, there's no legitimate reason someone would need to clip upvote/downvote dozens of times per seconds. It's not a video game.
I'm creating my source observable like this (make api call every 5s):
const obs$ = Observable.interval(5000).switchMap(() => makeApiCall());
And I want to modify $obs so that it has the following characteristics:
start the observable only when at there's at least 1 subscriber
multicast. I.e. if I obs$.subscribe(...) twice, the underlying code makeApiCall() should only run once.
any subscriber which subscribes at any time should have immediately the last emitted value (and not wait ~5s until the next value emits)
retryable. If one makeApiCall() errors, I want (if possible) all subscribers to get an error notification, but reconnect to $obs, and continue doing makeApiCall() every 5s
So far I found the following leads:
It seems like I'd need to create a BehaviorSubject myBehaviorSubject, do a single subscription obs$.subscribe(myBehaviorSubject), and any other observers should subscribe to myBehaviorSubject. Not sure if that answers the "retryable" part.
I also looked at shareReplay, seems like $obs.shareReplay(1) would do the trick (for the 4 requirements). If I understood correctly it subscribes a ReplaySubject(1) to the source observable, and future observers subscribe to this ReplaySubject. Is there an equivalent shareBehavior?
In RxSwift, I found shareReplayLatestWhileConnected, which seems like the shareBehavior I was imagining. But it doesn't exist in RxJS.
Any ideas what is the best way to achieve this?
As you mentioned, shareReplay(1) pretty much gets you there. It will multicast the response to current subscribers and replay the last value (if there is one) to new subscribers. That seems like what you would want rather than shareBehavior (if it existed) since you are calling an api and there isn't an initial value.
You should know that shareReplay will create a subscription to the source stream but will only unsubscribe when refCount === 0 AND the source stream terminates (error or complete). This means that after the first subscription that the interval will start and even when there are no more subscriptions it will continue.
If you want to stop the interval when no-one is subscribed then use multicast(new ReplaySubject(1)).refCount(). The multicast operator will create a single subscription to the source stream and push all values into the subject provided as an instance (multicast(new Subject())) or by the factory (multicast(() => new Subject())). All subscribers to the stream after the multicast will subscribe to the multicast subject. So when a value flows through the multicast operator all of its subscribers will get that value. You can change the type of subject that you pass to multicast to change its behavior. In your case you probably want a ReplaySubject so that it will replay the last value to a new subscriber. You could use a BehaviorSubject too if you felt that met your need.
Now the multicast operator is connectable meaning that you would have to call connect() on the stream to make it hot. The refCount operator basically makes a connectable observable act like an ordinary observable in that it will become hot when subscribed but will become cold when there are no subscribers. It does this be keeping an internal reference count (hence the name refCount). When refCount === 0 it will disconnect.
This is the same thing as shareReplay(1) with one minor but important difference which is that when there are no more subscribers that it will unsubscribe from the source stream. If you are using a factory method to create a new subject when subscribing to the source (ex: multicast(() => new ReplaySubject(1))) then you will lose your value when the stream goes from hot to cold to hot since it will create a new subject each time it goes hot. If you want to keep the same subject between source subscriptions then you can pass in a subject instead of a factory (ex: multicast(new ReplaySubject(1)) or use its alias publishReplay(1).
As far as your last requirement of providing errors to your subscribers and then resubscribing, you can't call the error callback on a subscription and then continue getting values on the next callback. An unhandled error will end a subscription if it reaches it. So you have to catch it before it gets there and turn it into a normal message if you want your subscription to see it and still live. You can do this like so: catch((err) => of(err)) and just flag it somehow. If you want to mute it then return empty().
If you want to retry immediately then you could use the retryWhen operator but you probably want to put that before the sharing operator to make it universal. However this also prevents your subscribers from knowing about an error. Since the root of your stream is an interval and the error came from the inner observable returned from the switchMap, the error will not kill the source of the stream but it could kill the subscription. So as long as you handle the error (catch/catchError) the api call will be retried on the next interval.
Also, you may want timer(0, 5000) instead of interval so that your api call immediately fires and then fires on a 5 second interval after that.
So I would suggest something like the following:
let count = 0;
function makeApiCall() {
return Rx.Observable.of(count++).delay(1000);
}
const obs$ = Rx.Observable.timer(0, 5000)
.switchMap(() => makeApiCall().catch(() => Rx.Observable.empty()))
.publishReplay(1)
.refCount();
console.log('1 subscribe');
let firstSub = obs$.subscribe((x) => { console.log('1', x); });
let secondSub;
let thirdSub;
setTimeout(() => {
console.log('2 subscribe');
secondSub = obs$.subscribe((x) => { console.log('2', x); });
}, 7500);
setTimeout(() => {
console.log('1 unsubscribe');
firstSub.unsubscribe();
console.log('2 unsubscribe');
secondSub.unsubscribe();
}, 12000);
setTimeout(() => {
console.log('3 subscribe');
thirdSub = obs$.subscribe((x) => { console.log('3', x); });
}, 17000);
setTimeout(() => {
console.log('3 unsubscribe');
thirdSub.unsubscribe();
}, 30000);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.10/Rx.min.js"></script>
For convenience, here are aliases for multicast:
publish() === multicast(new Subject())
publishReplay(#) === multicast(new ReplaySubject(#))
publishBehavior(value) === multicast(new BehaviorSubject(value))
I just tried to implement this with rxjs 6, but the implementation feels kinda hacky. I think there should be a much cleaner way to achieve this.
The expected behavior is:
As long as there are observers, they all get the same values.
When there are 0 observers, the source subscription is closed but the ReplaySubject is not completed.
When new observers subscribe again they get the last N values and a new subscription to source is established.
When the source completes or throws an error, current observers are completed resp. notified.
After source completion or source error, new subscribers don't get replayed values any more and are completed immediately.
export function shareReplayLatestWhileConnected<T>(count?: number) {
return function (source: Observable<T>): Observable<T> {
let done = false;
return source.pipe(
// Identify when source is completed or throws an error.
tap(
null,
() => (done = true),
() => (done = true),
),
multicast(
// Subject for multicasting
new ReplaySubject<T>(count),
// Selector function. Stop subscription on subject, when source is done, to kill all subscriptions.
(shared) => shared.pipe(takeWhile(() => !done)),
),
// I was not able to get rid of duplicate subscriptions. Multicast subscribed multiple times on the source.
share(),
);
};
}
Any tips on how I could improve this solution are very appreciated.
Use it like this:
const shared$ = source$.pipe(shareReplayLatestWhileConnected(1));