I have recurring data structure (observable) returned from backend it looks like:
[{
id:1,
userId: 111,
name: '',
children :[
{
id:3,
userId: 333,
name: '',
children: [...]
}
]
},
{
id:2,
userId:111,
name:'',
children: [...]
}]
I have another end point that returns user name by user id. I need to call this service wit each of IDs and map returned name to the structure. Is there any pretty solution to achieve this using RxJs operators ?
You can try an approach like the following. See comments inline for details.
// fetchStruct is a function that returns an Observable which notifies the initial structure
const struct$ = fetchStruct();
// here we start the RxJs pipe transformation
struct$.pipe(
// when struct$ emits the initial structure we pass the control to another observable chain
// this is done via the concatMap operator
concatMap(beStruct => { // beStruct is the structure returned by the back end
// from beStruct we construct an array of Observables
// fetchName is a function that returns an Observable that emits when the name is returned
arrObs = beStruct.map(el => fetchName(el.id))
// with forkJoin we execute all the Observables in the array in parallel
forkJoin(arrObs).pipe(
// forkJoin emits when all Observables have notified and it will emit
// an array of values with the same order as arrObs
// we can therefore loop through this array to enrich beStruct with the names
map(names => {
names.forEach((n, i) => beStruct[i].name = n);
return beStruct;
})
)
})
)
This is quite a typical case with RxJs. You may find some other frequent pattern in this blog.
First I'm going to assume that .children can be arrays of ids i.e. pointers to other users. (See example below.)
What I'd would do is transform the initial users array into an array of observables. Each doing a request to map an id to a name. Use merge(…) to fire off the requests.
Of course you don't want a million of HTTP requests going on at the same time so you can set the concurrency parameter of merge to a number that is appropriate to you. (In my example I set it to 2.)
Since you had an array in, you want an array out. You can simply accumulate into an array with the toArray operator.
In the example below:
How I would process the initial users array (i.e. .children should be pointers)
Map users into an array of observables to resolve the names
Merge that array of observables with a concurrency set to 2
Accumulate the result into an array
const get_name = user => fake_backend[user.id].pipe(map(name => ({...user, name})));
// ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^
// A B
//
// A: Simulate HTTP request latency to lookup an id and return a name
// B: Merge name into initial user object
merge(...users.map(get_name), 2).pipe(toArray()).subscribe(final_users => {
// ^ ^^^^^^^^^
// A B
//
// A: Concurrency. Maximum two HTTP requests at any one time.
// B: Accumulate each output into an array. Complete when merge is done.
console.log(final_users);
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.2.0/rxjs.umd.min.js" integrity="sha512-MlqMFvHwgWJ1vfts5fdC2WzxDaIXWfYuAd9Tb2lobtF61Gk+HIRDrbtxgasBSM9lZgOK9ilwK9LqFIYEV+k0IA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script>
const {merge, of} = rxjs;
const {tap, delay, map, toArray} = rxjs.operators;
const users =
[ { id: 1
, userId: 111
, name: ''
, children: [3]}
, { id: 2
, userId: 111
, name: ''
, children: [4]}
, { id: 3
, userId: 333
, name: ''
, children: []}
, { id: 4
, userId: 444
, name: ''
, children: []}];
const fake_backend =
{ 1: of('Foo').pipe(delay(3000), tap(() => console.log('user 1 ✓')))
, 2: of('Bar').pipe(delay(2000), tap(() => console.log('user 2 ✓')))
, 3: of('Baz').pipe(delay(1000), tap(() => console.log('user 3 ✓')))
, 4: of('Bat').pipe(delay(1000), tap(() => console.log('user 4 ✓'))) };
</script>
For your question here is a simple answer.
Steps:
From the list of Users end point you only need id's so you have to simplify the result from the users end point, instead of all the users data you only extract the id's from that end point and keep inside an array.
2.Convert the resultant array into another observable.
3.Send that into mergeMap and there you call a function with arguments to call your second URL service.
Here is a code snippet.
-//My first users service
let obsPattern1$ = this.http.get('https://reqres.in/api/users/');
obsPattern1$.pipe(
map(data=>data["data"].map(data=>{
return data.id
})),mergeMap(data=> data),mergeMap(data=>this.getUserFromHttp(data))
).subscribe(x=>console.log("final result",x))
}
//this is my second service function to fetch the individual item
getUserFromHttp(id) {
return this.http.get('https://reqres.in/api/users/' + id);
}
I'm trying to build a reusable piece of code for multi files upload.
I do not want to care about the HTTP layer implementation, I want to purely focus on the stream logic.
I've built the following function to mock the HTTP layer:
let fakeUploadCounter = 0;
const fakeUpload = () => {
const _fakeUploadCounter = ++fakeUploadCounter;
return from(
Array.from({ length: 100 })
.fill(null)
.map((_, i) => i)
).pipe(
mergeMap(x =>
of(x).pipe(
delay(x * 100),
switchMap(x =>
_fakeUploadCounter % 3 === 0 && x === 25
? throwError("Error happened!")
: of(x)
)
)
)
);
};
This function simulates the progress of the upload and the progress will fail at 25% of the upload every 3 files.
With this out of the way, let's focus on the important bit: The main stream.
Here's what I want to achieve:
Only use streams, no imperative programming, no tap to push a temporary result in a subject. I could build this. But I'm looking for an elegant solution
While some files are being uploaded, I want to be able to add more files to the upload queue
As a browser can deal with only 6 HTTP calls at the same time, I do not want to take too much of that amount and we should be able to upload only 3 files at the same time. As soon as one finishes or is stopped or throws, then another file should start
When a file upload throws, we should keep that file in the list of file and still display the progress. It won't increase anymore but at least the user gets to see where it failed. When that's the case, we should see some text on that row indicating that there was an error and a retry button should let us give another go at the upload or a discard button will let us remove it completely
Here's a visual explanation:
So far, here's the code I've got:
export class AppComponent {
public file$$: Subject<File> = new Subject();
public retryFile$$: Subject<File> = new Subject();
public stopFile$$: Subject<File> = new Subject();
public files$ = this.file$$.pipe(
mergeMap(file =>
this.retryFile$$.pipe(
filter(retryFile => retryFile === file),
startWith(null),
map(() =>
fakeUpload().pipe(
map(progress => ({ progress })),
takeUntil(
this.stopFile$$.pipe(filter(stopFile => stopFile === file))
),
catchError(() => of({ error: true })),
scan(
(acc, curr: { progress: number } | { error: true }) => ({
...acc,
...curr
}),
{
file,
progress: 0,
error: false
}
)
)
)
)
),
mergeAll(3), // 3 upload in parallel maximum
scan(
(acc, curr) => ({
...acc,
// todo we can't use the File reference directly here
// but we shouldn't use the file name either
// instead we should generate a unique ID for each upload
[curr.file.name]: curr
}),
{}
),
map(fileEntities => Object.values(fileEntities))
);
public addFile() {
this.file$$.next(new File([], `test-file-${filesCount}`));
filesCount++;
}
}
Here's the code in stackblitz that you can fork: https://stackblitz.com/edit/rxjs-upload-multiple-files-v2?file=src/app/app.component.ts
I'm pretty close! If you open the live demo in stackblitz on the right and click on the "Add file" button, you'll see that you can add many files and they'll all get uploaded. The 3rd one will fail gracefully.
Now what is not working how I'd like:
If you click quickly more than 3 times on the "add file" button, only 3 files will appear in the queue. I'd like to have all of them but only 3 should be uploading at the same time. Yet, all the files to be uploaded should be displayed in the view, just waiting to start
The stop button should remove any upload. Whether it's uploading or failed
Thanks for any help
Number 1:
If you click quickly more than 3 times on the "add file" button, only 3 files will appear in the queue. I'd like to have all of them but only 3 should be uploading at the same time. Yet, all the files to be uploaded should be displayed in the view, just waiting to start
First of all, this is a cool problem because as far as I could see, you can't simply compose the existing operators (Without getting stupid with partition). You need a custom operator that splits your stream. If you don't want to subscribe to your source twice, you should share before splitting.
There's quite a lot of work left to implement your solution the way you'd like. BUT, in terms of getting your stream to show all files regardless of whether they're currently loading, there's really just one piece missing.
You want to split your stream. One stream should emit default
{
file,
progress: 0,
error: false
}`
files right away and the second stream should emit updates to those files. The second stream will have mergeAll(3), but the first doesn't need this limitation as it's not making a network request. You merge these two-streams and either update or add new entries into your output as you see fit.
Here's an example of that at work. I made a dummy example to abstract away the implementation details a bit. I start out with an array of objects with this shape,
{
id: number,
message: "HeyThere" + id,
response: none
}
I make a fake httpRequest call that enriches an object to
{
id: number,
message: "HeyThere" + id,
response: "Hello"
}
The stream emits each time a new object is added or when an object is enriched. But the enriching stream is limited to max 3 httpRequest calls at once.
const httpRequest= () => {
return timer(4000).pipe(
map(_ => "Hello")
);
}
const arrayO = [];
arrayO.length = 10;
from(arrayO).pipe(
map((val, index) => ({
id: index,
message: "HeyThere" + index,
response: "None"
})),
share(),
s => merge(s, s.pipe(
map(ob => httpRequest().pipe(
map(val => ({...ob, response: val}))
)),
mergeAll(3)
)),
scan((acc, val: any) => {
acc.set(val.id, val);
return acc;
}, new Map<number, any>()),
debounceTime(250),
map(mapO => Array.from(mapO.values()))
).subscribe(console.log);
I added a debounce as I find it makes the output much easier to follow. Since I added all 10 un-enriched objects synchronously, it just spams 10 arrays to the output if I don't debounce. Also, since every fake HttpRequest takes exactly 4 seconds, I get three arrays spammed at the output every 4 seconds. Debounce stops the UI from stuttering or the console from getting spammed.
Number 2
The stop button should remove any upload. Whether it's uploading or failed
This is a can of worms because every canonical solution says you should make a state management system. That would be the easiest way to interact with files that are in Queue, Loading, Failed, and Loaded all in one uniform way.
It's pretty easy to implement a lightweight Redux-style state management system using RxJS (Just use scan to manage state and JSON objects representing events to transform state). The toughest part is managing your current httpRequests. You'd probably create a custom mergeAll() operator that takes in events, removes queued requests, and even cancels mid-flight requests if necessary.
Using a stopFile$$ works to cancel mid-flight requests but it'll fall apart if people want to stop a fileload that hasn't started yet (as per your first requirement, you want those vsible too). It's sort of brittle regardless because emiting on a suject never comes with the assurance that anybody is listening. Another reason that a redux-style management is the way to go.
This is a very interesting problem, here is my approach to it:
uploadFile$ = this.uploadFile.pipe(
multicast(new Subject<CustomFile>(), subject =>
merge(
subject.pipe(
mergeMap(
// `file.id` might be created with uuid() or something like that
(file, idx) =>
of({ status: FILE_STATUS.PENDING, ...file }).pipe(
observeOn(asyncScheduler),
takeUntil(subject)
)
)
),
subject.pipe(
mergeMap(
(file, idx) =>
fakeUpload(file).pipe(
map(progress => ({
...file,
progress,
status: FILE_STATUS.LOADING
})),
startWith({
name: file.name,
status: FILE_STATUS.LOADING,
id: file.id,
progress: 0
}),
catchError(() => of({ ...file, status: FILE_STATUS.FAILED })),
scan(
(acc, curr) => ({
...acc,
...curr
}),
{} as CustomFile
),
takeUntil(
this.stopFile.pipe(
tap(console.warn),
filter(f => f.id === file.id)
)
)
),
3
)
)
)
)
);
files$: Observable<CustomFile[]> = merge(
this.uploadFile$,
this.stopFile
).pipe(
tap(v =>
v.status === FILE_STATUS.REMOVED ? console.warn(v) : console.log(v)
),
scan((filesAcc, crtFile) => {
// if the file is being removed, we need to remove it from the list
if (crtFile.status === FILE_STATUS.REMOVED) {
const { [crtFile.id]: _, ...rest } = filesAcc;
return rest;
}
// simply return an updated copy of the object when the file has the status either
// * `pending`(the buffer's length is > 3)
// * `loading`(the file is being uploaded)
// * `failed`(an error occurred during the file upload, but we keep it in the list)
// * `retrying`(the `Retry` button has been pressed)
return {
...filesAcc,
[crtFile.id]: crtFile
};
}, Object.create(null)),
// Might want to replace this by making the `scan`'s seed return an object that implements a custom iterator
map(obj => Object.values(obj))
);
StackBlitz demo.
I think the biggest problem here was how to determine when the mergeMap's buffer is full, so that a pending item should be shown to the user. As you can see, I've solved this using the multicast's second parameter:
multicast(new Subject(), subject => ...)
multicast(new Subject), refCount(), without its second argument, it's the same as share(). But when you provide the second argument(a.k.a the selector), you can achieve some sort of local multicasting:
if (isFunction(selector)) {
return operate((source, subscriber) => {
// the first argument
const subject = subjectFactory();
/* .... */
selector(subject).subscribe(subscriber).add(source.subscribe(subject));
});
}
selector(subject).subscribe(subscriber) will subscribe to the observable(which can also be a Subject) returned from the selector. Then, with .add(source.subscribe(subject)), the source is subscribed to. In the selector, we've used merge(subject.pipe(...), subject.pipe(...)), each of which will gain access to what's being pushed into the stream. Because of add(source.subscribe(subject)), the source's value will be passed to the Subject instance, which has its subscribers.
So, the way I solved the aforementioned problem was to create a race between observables. The first contender is
// #1
subject.pipe(
mergeMap(
// `file.id` might be created with uuid() or something like that
(file, idx) =>
of({ status: FILE_STATUS.PENDING, ...file }).pipe(
observeOn(asyncScheduler),
takeUntil(subject)
)
)
),
and the second one is
// #2
subject.pipe(
mergeMap(
(file, dx) => fileUpload().pipe(
/* ... */
// emits synchronously - as soon as the inner subscriber is created
startWith(...)
)
)
)
So, as soon as the Subject(the subject variable in this case) receives the value from the source, it will send it to all of its subscribers - the 2 contenders. It all happens synchronously, which also means that the order matters. #1 will be the first subscriber to receive the value, and #2 will be second. The way the winner is selected is to see which one of the 2 subscribers emits first.
Notice that the first will pass along the value asynchronously(with the help of observeOn(asyncScheduler)) and the second one synchronously. The first one will emit first if the buffer is full, otherwise the second will emit.
I've ended up with the following:
export interface FileUpload {
file: File;
progress: number;
error: boolean;
toRemove: boolean;
}
export const uploadManager = () => {
const file$$: Subject<File> = new Subject();
const retryFile$$: Subject<File> = new Subject();
const stopFile$$: Subject<File> = new Subject();
const fileStartOrRetry$: Observable<File> = file$$.pipe(
mergeMap(file =>
retryFile$$.pipe(
filter(retryFile => retryFile === file),
startWith(file)
)
),
share()
);
const addFileToQueueAfterStartOrRetry$: Observable<
FileUpload
> = fileStartOrRetry$.pipe(
map(file => ({
file,
progress: 0,
error: false,
toRemove: false
}))
);
const markFileToBeRemovedAfterStop$: Observable<FileUpload> = stopFile$$.pipe(
map(file => ({
file,
progress: 0,
error: false,
toRemove: true
}))
);
const updateFileProgress$: Observable<FileUpload> = fileStartOrRetry$.pipe(
map(file =>
uploadMock().pipe(
map(progress => ({ progress })),
takeUntil(
stopFile$$.pipe(filter(stopFile => stopFile.name === file.name))
),
catchError(() => of({ error: true })),
scan(
(acc, curr: { progress: number } | { error: true }) => ({
...acc,
...curr
}),
{
file,
progress: 0,
error: false,
toRemove: false
}
)
)
),
// 3 upload in parallel maximum
mergeAll(3)
);
const files$: Observable<FileUpload[]> = merge(
addFileToQueueAfterStartOrRetry$,
updateFileProgress$,
markFileToBeRemovedAfterStop$
).pipe(
scan<FileUpload, { [key: string]: FileUpload }>((acc, curr) => {
if (curr.toRemove) {
const copy = { ...acc };
delete copy[curr.file.name];
return copy;
}
return {
...acc,
// todo we can't use the File reference directly here
// but we shouldn't use the file name either
// instead we should generate a unique ID for each upload
[curr.file.name]: curr
};
}, {}),
map(fileEntities => Object.values(fileEntities))
);
return {
files$,
file$$,
retryFile$$,
stopFile$$
};
};
It covers all the cases as demonstrated here: https://rxjs-upload-multiple-file-v3.stackblitz.io
The code is here: https://stackblitz.com/edit/rxjs-upload-multiple-file-v3?file=src/app/upload-manager.ts
It's based on Mrk Sef's suggestion. It clicked after he mentioned "You want to split your stream".
first func:
updateMark(item: MarkDTO) {
this.service
.put(item, this.resource)
.subscribe(() => this.markEdit = null);
}
second func:
put(item: MarkDTO, rcc: string): Observable<MarkDTO> {
const rdto = new MarkRDTO(item);
const url = `${this.getUrl('base')}${rcc}/marks/${rdto.rid}`;
const obs = this.http.put<MarkDTO>(url, rdto, { withCredentials: true })
.pipe(map((r: MarkDTO) => new MarkDTO(r)))
.share();
obs.subscribe(newMark => this.storage.update(newMark, rcc));
return obs;
}
in service i need to update data after request
but also in same time i need to clear current editItem
all of it must be done after i subscribe to one httpRequest
.share() - suport from rxjs-compat package (i want to remove this dep in closest time)
without .share() - work only 1 of 2 steps
current rxjs version is 6.3.3
Help who can...
There is a pipeable share operator, that you would use the same way you use map() (i.e. inside pipe()) and thus doesn't need rxjs-compat.
But you don't need share() here. All you need is the tap() operator:
put(item: MarkDTO, rcc: string): Observable<MarkDTO> {
const rdto = new MarkRDTO(item);
const url = `${this.getUrl('base')}${rcc}/marks/${rdto.rid}`;
return this.http.put<MarkDTO>(url, rdto, { withCredentials: true })
.pipe(
map(r => new MarkDTO(r)),
tap(newMark => this.storage.update(newMark, rcc))
);
}
This is for redux-observable but I think the general pattern is pretty generic to rxjs
I have a stream of events (from redux-observable, these are redux actions) and I'm specifically looking to pair up two differnt types of events for the same "resource" - "resource set active" and "resource loaded" - and emit a new event when these events "match up". The problem is these can come in in any order, for any resources, and can be fired multiple times. You might set something active before it is loaded, or load something before it is set active, and other resources might get loaded or set active in between.
What I want is a stream of "this resource, which is now loaded, is now active" - which also means that once a resource is loaded, it can be considered forever loaded.
If these events were not keyed by a resource id, then it would be very simple:
First I would split them up by type:
const setActive$ = action$.filter(a => a.type == 'set_active');
const load = action$.filter(a => a.type == 'loaded');
In a simple case where there is no keying, I'd say something like:
const readyevent$ = setActive$.withLatestFrom(loaded$)
then readyevent$ is just a stream of set_active events where there has been at least one loaded event.
But my problem is that the set_active and loaded events are each keyed by a resource id, and for reasons beyond my control, the property to identify the resource is different in the two events. So this becomes something like:
const setActive$ = action$.filter(a => a.type === 'set_active').groupBy(a => a.activeResourceId);
const loaded$ = action$.filter(a => a.type === 'loaded').groupBy(a => a.id);
but from this point I can't really figure out how to then "re-join
" these two streams-of-grouped-observables on the same key, so that I can emit a stream of withLatestFrom actions.
I believe this does what you are describing:
const action$ = Rx.Observable.from([
{ activeResourceId: 1, type: 'set_active' },
{ id: 2, type: 'loaded' },
{ id: 1, type: 'random' },
{ id: 1, type: 'loaded' },
{ activeResourceId: 2, type: 'set_active' }
]).zip(
Rx.Observable.interval(500),
(x) => x
).do((x) => { console.log('action', x); }).share();
const setActiveAction$ = action$.filter(a => a.type === 'set_active')
.map(a => a.activeResourceId)
.distinct();
const allActive$ = setActiveAction$.scan((acc, curr) => [...acc, curr], []);
const loadedAction$ = action$.filter(a => a.type === 'loaded')
.map(a => a.id)
.distinct();
const allLoaded$ = loadedAction$.scan((acc, curr) => [...acc, curr], []);
Rx.Observable.merge(
setActiveAction$
.withLatestFrom(allLoaded$)
.filter(([activeId, loaded]) => loaded.includes(activeId)),
loadedAction$
.withLatestFrom(allActive$)
.filter(([loadedId, active]) => active.includes(loadedId))
).map(([id]) => id)
.distinct()
.subscribe((id) => { console.log(`${id} is loaded and active`); });
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.10/Rx.min.js"></script>
The basic approach is to create a distinct stream for each action type and join it with the distinct aggregate of the other. Then merge the two streams. This will emit the value when there are matching setActive and loaded events. The distinct() on the end of the merge makes it so that you will only get notified once. If you want a notification on each setActive action after the initial one then just remove that operator.
groupBy looks somewhat complicated to do this with, there's a key value in there but you get an Observable of Observables - so maybe a little hard to get right.
I would map the id to a common property and then use scan to combine. I use this pattern for grouping in my app.
The accumulator in the scan is an object, which is used as an associative array - each property is an id and the property value is an array of actions accumulated so far.
After the scan, we convert to an observable stream of arrays of matching actions - sort of like withLatestFrom but some arrays may not yet be complete.
The next step is to filter for those arrays we consider complete.
Since you say
where there has been at least one loaded event
I'm going to assume that the presence of two or more actions, with at least one is type 'loaded' - but it's a bit tricky to tell from your question if that is sufficient.
Finally, reset that id in the accumulator as presumably it may occur again later in the stream.
const setActive$ = action$.filter(a => a.type === 'set_active')
.map(a => { return { id: a.activeResourceId, action: a } });
const loaded$ = action$.filter(a => a.type === 'loaded')
.map(a => { return { id: a.id, action: a } });
const accumulator = {};
const readyevent$: Observable<action[]> =
Observable.merge(setActive$, loaded$)
.scan((acc, curr) => {
acc[curr.id] = acc[curr.id] || [];
acc[curr.id].push(curr.action)
}, accumulator)
.mergeMap((grouped: {}) => Observable.from(
Object.keys(grouped).map(key => grouped[key])
))
.filter((actions: action[]) => {
return actions.length > 1 && actions.some(a => a.type === 'loaded')
})
.do(actions => {
const id = actions.find(a => a.type === 'loaded').id;
accumulator[id] = [];
});
I'm working with RxJs and I have to make a polling mechanism to retrieve updates from a server.
I need to make a request every second, parse the updates, emit it and remember its id, because I need it to request the next pack of updates like getUpdate(lastId + 1).
The first part is easy so I just use interval with mergeMap
let lastId = 0
const updates = Rx.Observable.interval(1000)
.map(() => lastId)
.mergeMap((offset) => getUpdates(offset + 1))
I'm collecting identifiers like this:
updates.pluck('update_id').scan(Math.max, 0).subscribe(val => lastId = val)
But this solution isn't pure reactive and I'm looking for the way to omit the usage of "global" variable.
How can I improve the code while still being able to return observable containing just updates to the caller?
UPD.
The server response for getUpdates(id) looks like this:
[
{ update_id: 1, payload: { ... } },
{ update_id: 3, payload: { ... } },
{ update_id: 2, payload: { ... } }
]
It may contain 0 to Infinity updates in any order
Something like this? Note that this is an infinite stream since there is no condition to abort; you didn't give one.
// Just returns the ID as the update_id.
const fakeResponse = id => {
return [{ update_id: id }];
};
// Fakes the actual HTTP call with a network delay.
const getUpdates = id => Rx.Observable.of(null).delay(250).map(() => fakeResponse(id));
// Start with update_id = 0, then recursively call with the last
// returned ID incremented by 1.
// The actual emissions on this stream will be the full server responses.
const updates$ = getUpdates(0)
.expand(response => Rx.Observable.of(null)
.delay(1000)
.switchMap(() => {
const highestId = Math.max(...response.map(update => update.update_id));
return getUpdates(highestId + 1);
})
)
updates$.take(5).subscribe(console.log);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.6/Rx.js"></script>
To define the termination of the stream, you probably want to hook into the switchMap at the end; use whatever property of response to conditionally return Observable.empty() instead of calling getUpdates again.