How to queue actions through time properly when using redux-observables - rxjs

I'm trying to create an Epic that queues actions to be dispatched, in this case, the actions are messages to be displayed with a snackbar so that, for example, three errors happened almost simultaneously, and I want to display the three error messages with a SnackBar, the Epic should dispatch an action to display the first one for 3 seconds, then display the second one for 3 seconds and then the third one for 3 seconds.
Also if one of the snackbars gets closed, the second one in the "queue" should be displayed and so on. What I'm doing at the moment is dispatching the action that displays the message in Epics that catch the errors, then I'm creating Epics dispatch the action to close the snackbar after 3 seconds of delay, with the action CLOSE_SNACKBAR. Which methods should I use in order to achieve this?
A basic Epic I've implemented looks like this, basically the action that changes the state from the snackbar to open and displays the error message is dispatched from another epic, the one that catches the error, and then another Epic dispatches the action to close the snackbar after 3 seconds, but I haven't figured out how to do this queueing kind of action so that each message can be displayed for 3 seconds each, right now if an error occurs right after the first one, the second message will be displayed without waiting the 3 seconds after the previous message.
Here are some basic examples of my epics (Ignore de request part, ):
const getUserEpic = (action$, store) => (
action$.ofType(actionTypes.DATA_REQUESTED)
.switchMap(action => {
const { orderData, queryData, pagerData } = store.getState().usuario.list;
const params = constructParams(queryData, pagerData, orderData);
return Observable.defer(() => axios.get(`users`, { params }))
.retry(NETWORK.RETRIES).mergeMap(response => {
Observable.of(actions.dataRequestSucceeded(response.data.rows))
}).catch(error => Observable.concat(
Observable.of(actions.dataRequestFailed(error)),
Observable.of({
type:'DISPLAY_DATA_REQUEST_FAILED_MESSAGE',
open:true,
message:'Failed to get Users Data'
})
));
})
)
const getRoleEpic = (action$, store) => (
action$.ofType(actionTypes.DATA_REQUESTED)
.switchMap(action => {
const { orderData, queryData, pagerData } = store.getState().usuario.list;
const params = constructParams(queryData, pagerData, orderData);
return Observable.defer(() => axios.get(`role`, { params }))
.retry(NETWORK.RETRIES).mergeMap(response => {
Observable.of(actions.dataRequestSucceeded(response.data.rows))
}).catch(error => Observable.concat(
Observable.of(actions.dataRequestFailed(error)),
Observable.of({
type:'DISPLAY_DATA_REQUEST_FAILED_MESSAGE',
open:true,
message:'Failed to get Roles Data'
})
));
})
)
This two epics do basically the seme, they are doing a GET request to the backend, but if they fail, they dispatch the action that opens the snackbar with an error message, and both error messages are different.
And this one is the epic that currently closes the Snackbar after 3 seconds:
const displayDataRequestFailedEpic = (action$, store) => (
action$.ofType(actionTypes.DISPLAY_DATA_REQUEST_FAILED)
.debounceTime(3e3)
.switchMap( action => {
return Observable.of({
type:'CLOSE_SNACKBAR',
open:false,
message:''
})
})
)
Imagine I do both requests really fast and both of them fail. I want show all the errors that happened, one after another for 3 seconds each,

This hasn't been tested...
First, need to separate the action that displays messages from the action that is causing the display:
const getUserEpic = (action$, store) => (
action$.ofType(actionTypes.DATA_REQUESTED)
.switchMap(action => {
const { orderData, queryData, pagerData } = store.getState().usuario.list;
const params = constructParams(queryData, pagerData, orderData);
return Observable.defer(() => axios.get(`users`, { params }))
.retry(NETWORK.RETRIES).mergeMap(response => {
return Observable.of(actions.dataRequestSucceeded(response.data.rows))
}).catch(error => Observable.of(actions.dateRequestFailed(error))
// }).catch(error => Observable.concat(
// Observable.of(actions.dataRequestFailed(error)),
//
// this will be handled elsewhere
//
// Observable.of({
// type:'DISPLAY_DATA_REQUEST_FAILED_MESSAGE',
// open:true,
// message:'Failed to get Users Data'
// })
));
})
)
next, need to define some method that creates the DISPLAY_MESSAGE action from whatever action is triggering the message. Included in the action should be a unique id used to track which message needs to be closed.
Finally, use concatMap() to create and destroy the messages.
const displayEpic = action$ => action$
.ofType(actionTypes.DATA_REQUEST_FAILED)
.concatMap(action => {
const messageAction = actions.displayMessage(action);
const { id } = messageAction;
return Rx.Observable.merge(
Rx.Observable.of(messageAction),
Rx.Observable.of(actions.closeMessage(id))
.delay(3000)
.takeUntil(action$
.ofType(actionTypes.CLOSE_MESSAGE)
.filter(a => a.id === id))
);
});

Related

Rxjs multi file upload queue with retry and stop behavior

I'm trying to build a reusable piece of code for multi files upload.
I do not want to care about the HTTP layer implementation, I want to purely focus on the stream logic.
I've built the following function to mock the HTTP layer:
let fakeUploadCounter = 0;
const fakeUpload = () => {
const _fakeUploadCounter = ++fakeUploadCounter;
return from(
Array.from({ length: 100 })
.fill(null)
.map((_, i) => i)
).pipe(
mergeMap(x =>
of(x).pipe(
delay(x * 100),
switchMap(x =>
_fakeUploadCounter % 3 === 0 && x === 25
? throwError("Error happened!")
: of(x)
)
)
)
);
};
This function simulates the progress of the upload and the progress will fail at 25% of the upload every 3 files.
With this out of the way, let's focus on the important bit: The main stream.
Here's what I want to achieve:
Only use streams, no imperative programming, no tap to push a temporary result in a subject. I could build this. But I'm looking for an elegant solution
While some files are being uploaded, I want to be able to add more files to the upload queue
As a browser can deal with only 6 HTTP calls at the same time, I do not want to take too much of that amount and we should be able to upload only 3 files at the same time. As soon as one finishes or is stopped or throws, then another file should start
When a file upload throws, we should keep that file in the list of file and still display the progress. It won't increase anymore but at least the user gets to see where it failed. When that's the case, we should see some text on that row indicating that there was an error and a retry button should let us give another go at the upload or a discard button will let us remove it completely
Here's a visual explanation:
So far, here's the code I've got:
export class AppComponent {
public file$$: Subject<File> = new Subject();
public retryFile$$: Subject<File> = new Subject();
public stopFile$$: Subject<File> = new Subject();
public files$ = this.file$$.pipe(
mergeMap(file =>
this.retryFile$$.pipe(
filter(retryFile => retryFile === file),
startWith(null),
map(() =>
fakeUpload().pipe(
map(progress => ({ progress })),
takeUntil(
this.stopFile$$.pipe(filter(stopFile => stopFile === file))
),
catchError(() => of({ error: true })),
scan(
(acc, curr: { progress: number } | { error: true }) => ({
...acc,
...curr
}),
{
file,
progress: 0,
error: false
}
)
)
)
)
),
mergeAll(3), // 3 upload in parallel maximum
scan(
(acc, curr) => ({
...acc,
// todo we can't use the File reference directly here
// but we shouldn't use the file name either
// instead we should generate a unique ID for each upload
[curr.file.name]: curr
}),
{}
),
map(fileEntities => Object.values(fileEntities))
);
public addFile() {
this.file$$.next(new File([], `test-file-${filesCount}`));
filesCount++;
}
}
Here's the code in stackblitz that you can fork: https://stackblitz.com/edit/rxjs-upload-multiple-files-v2?file=src/app/app.component.ts
I'm pretty close! If you open the live demo in stackblitz on the right and click on the "Add file" button, you'll see that you can add many files and they'll all get uploaded. The 3rd one will fail gracefully.
Now what is not working how I'd like:
If you click quickly more than 3 times on the "add file" button, only 3 files will appear in the queue. I'd like to have all of them but only 3 should be uploading at the same time. Yet, all the files to be uploaded should be displayed in the view, just waiting to start
The stop button should remove any upload. Whether it's uploading or failed
Thanks for any help
Number 1:
If you click quickly more than 3 times on the "add file" button, only 3 files will appear in the queue. I'd like to have all of them but only 3 should be uploading at the same time. Yet, all the files to be uploaded should be displayed in the view, just waiting to start
First of all, this is a cool problem because as far as I could see, you can't simply compose the existing operators (Without getting stupid with partition). You need a custom operator that splits your stream. If you don't want to subscribe to your source twice, you should share before splitting.
There's quite a lot of work left to implement your solution the way you'd like. BUT, in terms of getting your stream to show all files regardless of whether they're currently loading, there's really just one piece missing.
You want to split your stream. One stream should emit default
{
file,
progress: 0,
error: false
}`
files right away and the second stream should emit updates to those files. The second stream will have mergeAll(3), but the first doesn't need this limitation as it's not making a network request. You merge these two-streams and either update or add new entries into your output as you see fit.
Here's an example of that at work. I made a dummy example to abstract away the implementation details a bit. I start out with an array of objects with this shape,
{
id: number,
message: "HeyThere" + id,
response: none
}
I make a fake httpRequest call that enriches an object to
{
id: number,
message: "HeyThere" + id,
response: "Hello"
}
The stream emits each time a new object is added or when an object is enriched. But the enriching stream is limited to max 3 httpRequest calls at once.
const httpRequest= () => {
return timer(4000).pipe(
map(_ => "Hello")
);
}
const arrayO = [];
arrayO.length = 10;
from(arrayO).pipe(
map((val, index) => ({
id: index,
message: "HeyThere" + index,
response: "None"
})),
share(),
s => merge(s, s.pipe(
map(ob => httpRequest().pipe(
map(val => ({...ob, response: val}))
)),
mergeAll(3)
)),
scan((acc, val: any) => {
acc.set(val.id, val);
return acc;
}, new Map<number, any>()),
debounceTime(250),
map(mapO => Array.from(mapO.values()))
).subscribe(console.log);
I added a debounce as I find it makes the output much easier to follow. Since I added all 10 un-enriched objects synchronously, it just spams 10 arrays to the output if I don't debounce. Also, since every fake HttpRequest takes exactly 4 seconds, I get three arrays spammed at the output every 4 seconds. Debounce stops the UI from stuttering or the console from getting spammed.
Number 2
The stop button should remove any upload. Whether it's uploading or failed
This is a can of worms because every canonical solution says you should make a state management system. That would be the easiest way to interact with files that are in Queue, Loading, Failed, and Loaded all in one uniform way.
It's pretty easy to implement a lightweight Redux-style state management system using RxJS (Just use scan to manage state and JSON objects representing events to transform state). The toughest part is managing your current httpRequests. You'd probably create a custom mergeAll() operator that takes in events, removes queued requests, and even cancels mid-flight requests if necessary.
Using a stopFile$$ works to cancel mid-flight requests but it'll fall apart if people want to stop a fileload that hasn't started yet (as per your first requirement, you want those vsible too). It's sort of brittle regardless because emiting on a suject never comes with the assurance that anybody is listening. Another reason that a redux-style management is the way to go.
This is a very interesting problem, here is my approach to it:
uploadFile$ = this.uploadFile.pipe(
multicast(new Subject<CustomFile>(), subject =>
merge(
subject.pipe(
mergeMap(
// `file.id` might be created with uuid() or something like that
(file, idx) =>
of({ status: FILE_STATUS.PENDING, ...file }).pipe(
observeOn(asyncScheduler),
takeUntil(subject)
)
)
),
subject.pipe(
mergeMap(
(file, idx) =>
fakeUpload(file).pipe(
map(progress => ({
...file,
progress,
status: FILE_STATUS.LOADING
})),
startWith({
name: file.name,
status: FILE_STATUS.LOADING,
id: file.id,
progress: 0
}),
catchError(() => of({ ...file, status: FILE_STATUS.FAILED })),
scan(
(acc, curr) => ({
...acc,
...curr
}),
{} as CustomFile
),
takeUntil(
this.stopFile.pipe(
tap(console.warn),
filter(f => f.id === file.id)
)
)
),
3
)
)
)
)
);
files$: Observable<CustomFile[]> = merge(
this.uploadFile$,
this.stopFile
).pipe(
tap(v =>
v.status === FILE_STATUS.REMOVED ? console.warn(v) : console.log(v)
),
scan((filesAcc, crtFile) => {
// if the file is being removed, we need to remove it from the list
if (crtFile.status === FILE_STATUS.REMOVED) {
const { [crtFile.id]: _, ...rest } = filesAcc;
return rest;
}
// simply return an updated copy of the object when the file has the status either
// * `pending`(the buffer's length is > 3)
// * `loading`(the file is being uploaded)
// * `failed`(an error occurred during the file upload, but we keep it in the list)
// * `retrying`(the `Retry` button has been pressed)
return {
...filesAcc,
[crtFile.id]: crtFile
};
}, Object.create(null)),
// Might want to replace this by making the `scan`'s seed return an object that implements a custom iterator
map(obj => Object.values(obj))
);
StackBlitz demo.
I think the biggest problem here was how to determine when the mergeMap's buffer is full, so that a pending item should be shown to the user. As you can see, I've solved this using the multicast's second parameter:
multicast(new Subject(), subject => ...)
multicast(new Subject), refCount(), without its second argument, it's the same as share(). But when you provide the second argument(a.k.a the selector), you can achieve some sort of local multicasting:
if (isFunction(selector)) {
return operate((source, subscriber) => {
// the first argument
const subject = subjectFactory();
/* .... */
selector(subject).subscribe(subscriber).add(source.subscribe(subject));
});
}
selector(subject).subscribe(subscriber) will subscribe to the observable(which can also be a Subject) returned from the selector. Then, with .add(source.subscribe(subject)), the source is subscribed to. In the selector, we've used merge(subject.pipe(...), subject.pipe(...)), each of which will gain access to what's being pushed into the stream. Because of add(source.subscribe(subject)), the source's value will be passed to the Subject instance, which has its subscribers.
So, the way I solved the aforementioned problem was to create a race between observables. The first contender is
// #1
subject.pipe(
mergeMap(
// `file.id` might be created with uuid() or something like that
(file, idx) =>
of({ status: FILE_STATUS.PENDING, ...file }).pipe(
observeOn(asyncScheduler),
takeUntil(subject)
)
)
),
and the second one is
// #2
subject.pipe(
mergeMap(
(file, dx) => fileUpload().pipe(
/* ... */
// emits synchronously - as soon as the inner subscriber is created
startWith(...)
)
)
)
So, as soon as the Subject(the subject variable in this case) receives the value from the source, it will send it to all of its subscribers - the 2 contenders. It all happens synchronously, which also means that the order matters. #1 will be the first subscriber to receive the value, and #2 will be second. The way the winner is selected is to see which one of the 2 subscribers emits first.
Notice that the first will pass along the value asynchronously(with the help of observeOn(asyncScheduler)) and the second one synchronously. The first one will emit first if the buffer is full, otherwise the second will emit.
I've ended up with the following:
export interface FileUpload {
file: File;
progress: number;
error: boolean;
toRemove: boolean;
}
export const uploadManager = () => {
const file$$: Subject<File> = new Subject();
const retryFile$$: Subject<File> = new Subject();
const stopFile$$: Subject<File> = new Subject();
const fileStartOrRetry$: Observable<File> = file$$.pipe(
mergeMap(file =>
retryFile$$.pipe(
filter(retryFile => retryFile === file),
startWith(file)
)
),
share()
);
const addFileToQueueAfterStartOrRetry$: Observable<
FileUpload
> = fileStartOrRetry$.pipe(
map(file => ({
file,
progress: 0,
error: false,
toRemove: false
}))
);
const markFileToBeRemovedAfterStop$: Observable<FileUpload> = stopFile$$.pipe(
map(file => ({
file,
progress: 0,
error: false,
toRemove: true
}))
);
const updateFileProgress$: Observable<FileUpload> = fileStartOrRetry$.pipe(
map(file =>
uploadMock().pipe(
map(progress => ({ progress })),
takeUntil(
stopFile$$.pipe(filter(stopFile => stopFile.name === file.name))
),
catchError(() => of({ error: true })),
scan(
(acc, curr: { progress: number } | { error: true }) => ({
...acc,
...curr
}),
{
file,
progress: 0,
error: false,
toRemove: false
}
)
)
),
// 3 upload in parallel maximum
mergeAll(3)
);
const files$: Observable<FileUpload[]> = merge(
addFileToQueueAfterStartOrRetry$,
updateFileProgress$,
markFileToBeRemovedAfterStop$
).pipe(
scan<FileUpload, { [key: string]: FileUpload }>((acc, curr) => {
if (curr.toRemove) {
const copy = { ...acc };
delete copy[curr.file.name];
return copy;
}
return {
...acc,
// todo we can't use the File reference directly here
// but we shouldn't use the file name either
// instead we should generate a unique ID for each upload
[curr.file.name]: curr
};
}, {}),
map(fileEntities => Object.values(fileEntities))
);
return {
files$,
file$$,
retryFile$$,
stopFile$$
};
};
It covers all the cases as demonstrated here: https://rxjs-upload-multiple-file-v3.stackblitz.io
The code is here: https://stackblitz.com/edit/rxjs-upload-multiple-file-v3?file=src/app/upload-manager.ts
It's based on Mrk Sef's suggestion. It clicked after he mentioned "You want to split your stream".

how to use multicasting obs with behavioursubject?

In general we need behavior subject functionality. But only on first subscription we should send subscribe to server in REST. And to send unsubscribe on the last unsubscribe, and all late observers subscribed will gwt the latest json recwived from the first. can i do it using rxjs operaTors and how? or shoul i use custom obserbale ?
currently the custom code for this is this:
public observable: Observable<TPattern> = new Observable((observer: Observer<TPattern>) => {
this._observers.push(observer);
if (this._observers.length === 1) {
this._subscription = this.httpRequestStream$
.pipe(
map((jsonObj: any) => {
this._pattern = jsonObj.Data;
return this._pattern;
})
)
.subscribe(
(data) => this._observers.forEach((obs) => obs.next(data)),
(error) => this._observers.forEach((obs) => obs.error(error)),
() => this._observers.forEach((obs) => obs.complete())
);
}
if (this._pattern !== null) {
observer.next(this._pattern); // send last updated array
}
return () => {
const index: number = this._observers.findIndex((element) => element === observer);
this._observers.splice(index, 1);
if (this._observers.length === 0) {
this._subscription.unsubscribe();
this._pattern = null; // clear pattern when unsubscribed
}
};
});
Sounds like you need a shareReplay(1), it will share the latest response with all subscribes.
const stream$ = httpRequestStream$.pipe(
shareReplay(1),
),
stream$.subscribe(); // sends the request and gets its result
stream$.subscribe(); // doesn't send it but gets cached result
stream$.subscribe(); // doesn't send it but gets cached result
stream$.subscribe(); // doesn't send it but gets cached result

Intercept each request with before and after hooks

I have to make 5 requests (order doesn't matter) to 5 different endpoints. The URL of these endpoints is the same, except for the business line. These business lines are the array of the from.
I want show a skeleton loader before each request and hide once it finish. So, basically the flow is:
1. [Hook - before request]
2. [Log of data fetched]
3. [Hook - after request]
This is my service:
export function getInsurances(
userIdentity: string,
hooks?: RequestHooks
): Observable<Policy[]> {
return from(["all", "vehicle", "health", "soat", "plans"]).pipe(
tap(() => hooks?.beforeRequest && hooks.beforeRequest()),
flatMap<string, Observable<Policy[]>>(businessLine => {
return InsurancesApi.getPolicies<Policy>(
userIdentity,
businessLine
).pipe(
map(policies => {
return policies.map(policy => PolicyStandarizer(policy));
}),
finalize(() => {
hooks?.afterRequest && hooks.afterRequest();
})
);
}),
catchError(err => of(err)),
takeUntil(HttpCancelator)
);
}
This is my subscribe:
const hooks = {
beforeRequest() {
Log.info("Before Request");
setStatus(HttpRequestStatus.PENDING);
},
afterRequest() {
Log.warn("After Request");
setStatus(HttpRequestStatus.RESOLVED);
},
};
getInsurances(userIdentity, hooks).subscribe(
policies => {
Log.normal("Policies:", policies);
setInsurances(policies);
},
(err: Error) => {
setError(err);
}
);
And have this output (sorry for paste the link, I can't embed the image because rep):
https://i.stack.imgur.com/Nbq49.png
The finalize is working fine, but the tap is executing five times at once.
Thank you.
I think you get this behavior because from emits the items synchronously, so its essentially the same as doing:
for (let i = 0; i < arr.length; i++) {
console.log('before req');
observer.next(arr[i]);
}
observer.complete();
afterRequest is shown properly because the actions involved are asynchronous.
If you want to trigger that event only once, before all the requests are fired, you could try this:
from([...])
.pipe(
finalize(() => hooks?.beforeRequest && hooks.beforeRequest()),
flatMap(/* ... */)
)
EDIT - log event before each request
flatMap(
value => concat(
of(null).pipe(
tap(() => hooks?.beforeRequest && hooks.beforeRequest()),
ignoreElements(), // Not interested in this observable's values
),
InsurancesApi.getPolicies(/* ... */)
)
)

RxJS throttle for AJAX requests

I want to create a function that will make AJAX requests to backend. And if this function is called many times at the same time, then it should not make many identical requests to the server. It must make only 1 request.
For example:
doAJAX('http://example-1.com/').subscribe(res => console.log); // must send a request
doAJAX('http://example-1.com/').subscribe(res => console.log); // must NOT send a request
doAJAX('http://example-2.com/').subscribe(res => console.log); // must send a request, bacause of different URL
window.setTimeout(() => {
doAJAX('http://example-2.com/').subscribe(res => console.log); // must send a request because too much time has passed since the last request
}, 3000)
All function calls should return a result, as if the request was actually made.
I think for this purpose I can use RxJS library.
I have done this:
const request$ = new Subject < string > ();
const response$ = request.pipe(
groupBy((url: string) => url),
flatMap(group => group.pipe(auditTime(500))), // make a request no more than once every 500 msec
map((url: string) => [
url,
from(fetch(url))
]),
share()
);
const doAJAX = (url: string): Observable <any> {
return new Observable(observe => {
response$
.pipe(
filter(result => result[0] === url),
first(),
flatMap(result => result[1])
)
.subscribe(
(response: any) => {
observe.next(response);
observe.complete();
},
err => {
observe.error(err);
}
);
request$.next(url);
});
}
I create request$ subject and response$ observable. doAjax function subscribes for response$ and send URL string to request$ subject. Also there are groupBy and auditTime operators in request$ stream. And filter operator in doAJAX function.
This code works but I think it is very difficult. Is there a way to make this task easier? Maybe RxJS scheduler or not use RxJS library at all
As the whole point of this is to memoize Http results and delay repeated calls, you might consider your own memoization. Example:
const memoise = (func) => {
let cache: { [key:string]: Observable<any> } = {};
return (...args): Observable<any> => {
const cacheKey = JSON.stringify(args)
cache[cacheKey] = cache[cacheKey] || func(...args).pipe(share());
return cache[cacheKey].pipe(
tap(() => timer(1000).subscribe(() => delete cache[cacheKey]))
);
}
}
Here is a Stackblitz DEMO

Wait for multiple actions to finish in epic before dispatching action

I have a component where I dispatch multiple actions:
dispatch(action1());
dispatch(action2());
I have an epic which listens for both of those actions, and makes an ajax call for each:
export const fetchObservable = action$ =>
action$.ofType(...actions) // wait for either action
.mergeMap(({ url, method, body, success, error, responseType }) =>
authenticate$
.mergeMap(token => ajax$(url, method, body, token, responseType))
.pluck('response')
.mergeMap(response => Observable.of(success(response), hideLoading()))
.startWith(showLoading())
.catch(response => Observable.of(error(response)))
);
I show a loading spinner with showLoading() and hide it with hideLoading()
The problem is that as soon as the first observable completes and Observable.of(success(response), hideLoading()) is called, the loading spinner disappears.
The reducer for the loader looks like this:
const loader = (state = false, action) => {
switch (action.type) {
case SHOW_LOADING:
return true;
case HIDE_LOADING:
return false;
default:
return state;
}
}
How can I wait for both actions to finish before calling hideLoading? How would I develop a solution that waits for x number of dispatches in a component?

Resources