I have a stream of events coming through via fromEventPattern like so:
fromEventPattern<IPsEvent>(addEventHandler).subscribe(ps$);
Due to business quirks, I expect that I will sometimes get an exception thrown, at which point I want to queue up the events and refire once that error state is resolved.
I've been trying the solution from Pausable buffer with RxJS to no avail. I am thinking it's because they are able to toggle through a separate observable whereas this is kind of asking to pause itself midstream. In the linked example I have blockingCallsAllowed$ rather than autoSave$. Here is my latest try:
const source$ = new Subject<IPsEvent>();
const blockingCallsAllowed$ = new BehaviorSubject(true);
const on$ = blockingCallsAllowed$.pipe(filter((v) => v));
const off$ = blockingCallsAllowed$.pipe(filter((v) => !v));
source$
.pipe(
map(() => {
try {
// line will throw exception at certain times
myFunction();
return true;
} catch (e) {
const i = setInterval(() => {
try {
myFunction();
console.log('good again');
blockingCallsAllowed$.next(true);
clearInterval(i);
} catch (er) {
// still in flux
}
}, 50);
return false;
}
}),
)
.subscribe(blockingCallsAllowed$);
const output$ = merge(
source$.pipe(bufferToggle(off$, () => on$)),
source$.pipe(windowToggle(on$, () => off$)),
).pipe(concatMap(from));
output$.subscribe((evt) => {
console.log('After buffers', evt);
});
// Add events from the Ps API to the event stream
fromEventPattern(addEventHandler).subscribe(source$);
Everything fires fine until the first exception and then it never outputs what it had buffered away, even though it fires that things are good again in console.log.
I am thinking there is some timing issue around relying on source$.pipe in the same execution and then the interval running later with .next. Can't nail it though after many different permutations of this code.
It's not clear to me what you're trying to implement. Though if you want to keep retrying myFunction() every 50ms until it succeeds and stop processing other events while this happens, concatMap basically does all that for you.
It will buffer emissions from the source while it waits for the inner observable to complete.
So what you're after might look like this:
source$.pipe(
concatMap(_ => of(true).pipe(
tap(_ => myFunction()),
retryWhen(errors => errors.pipe(
delay(50)
))
))
).subscribe();
I would like to execute code when the observable complete. In my code, i execute this:
compact(): Observable<FileManifest> {
return this.loadIndex().pipe(
mergeMap((index) => index.walk()),
map((entry) => entry.manifest),
notUndefined(),
writeAllMessages(this.newPath, ProtoFileManifest),
finalize(async () => {
await Promise.all([
promises.rm(this.journalPath, { force: true }),
promises.rm(this.manifestPath, { force: true }),
]);
await promises.rename(this.newPath, this.manifestPath);
}),
);
}
The problem is that the finalize method is made for synchronous code. When i execute asynchronous code like above, the code will be executed independently from the subscribe.
I would like this will be execute when disposing resource of the observable but i want that when i subscribe, i always receive the event.
How can i put asynchronous code in the finalize method ?
Thanks
Ulrich
One way to do it is to create three observables instead of trying to do it all
in one. Each will make up a link in the sequential async chain you want to
make.
In order for the side effects in the promise-based observables to be lazy, we use defer.
Note that the defer callback's return value can be an observable, or an
"ObservableInput", which is what RxJS calls values it knows how to turn
into observables. This value can be (among other things) a promise.
({
compact(): Observable<FileManifest> {
const writeToTempManifest$ = this.loadIndex().pipe(
mergeMap((index) => index.walk()),
map((entry) => entry.manifest),
notUndefined(),
writeAllMessages(this.newPath, ProtoFileManifest)
);
const removeOldManifest$ = defer(() =>
Promise.all([
promises.rm(this.journalPath, { force: true }),
promises.rm(this.manifestPath, { force: true }),
])
);
const renameNewManifest$ = defer(() =>
promises.rename(this.newPath, this.manifestPath)
);
return from([
writeToTempManifest$,
removeOldManifest$,
renameNewManifest$,
]).pipe(concatAll());
},
});
Note that each of these observables potentially emits something (though I'm not familiar with the API). The first emits whatever the writeAllMessages operator does, while the second and third emit the resolved values of their respective promises. In the case of the second one, that's a two element array from the Promise.all.
If you want to suppress an observable's emitted values while still keeping it open until it completes, you can create an operator that does just that:
const silence = pipe(concatMapTo(EMPTY));
I'm trying to build a reusable piece of code for multi files upload.
I do not want to care about the HTTP layer implementation, I want to purely focus on the stream logic.
I've built the following function to mock the HTTP layer:
let fakeUploadCounter = 0;
const fakeUpload = () => {
const _fakeUploadCounter = ++fakeUploadCounter;
return from(
Array.from({ length: 100 })
.fill(null)
.map((_, i) => i)
).pipe(
mergeMap(x =>
of(x).pipe(
delay(x * 100),
switchMap(x =>
_fakeUploadCounter % 3 === 0 && x === 25
? throwError("Error happened!")
: of(x)
)
)
)
);
};
This function simulates the progress of the upload and the progress will fail at 25% of the upload every 3 files.
With this out of the way, let's focus on the important bit: The main stream.
Here's what I want to achieve:
Only use streams, no imperative programming, no tap to push a temporary result in a subject. I could build this. But I'm looking for an elegant solution
While some files are being uploaded, I want to be able to add more files to the upload queue
As a browser can deal with only 6 HTTP calls at the same time, I do not want to take too much of that amount and we should be able to upload only 3 files at the same time. As soon as one finishes or is stopped or throws, then another file should start
When a file upload throws, we should keep that file in the list of file and still display the progress. It won't increase anymore but at least the user gets to see where it failed. When that's the case, we should see some text on that row indicating that there was an error and a retry button should let us give another go at the upload or a discard button will let us remove it completely
Here's a visual explanation:
So far, here's the code I've got:
export class AppComponent {
public file$$: Subject<File> = new Subject();
public retryFile$$: Subject<File> = new Subject();
public stopFile$$: Subject<File> = new Subject();
public files$ = this.file$$.pipe(
mergeMap(file =>
this.retryFile$$.pipe(
filter(retryFile => retryFile === file),
startWith(null),
map(() =>
fakeUpload().pipe(
map(progress => ({ progress })),
takeUntil(
this.stopFile$$.pipe(filter(stopFile => stopFile === file))
),
catchError(() => of({ error: true })),
scan(
(acc, curr: { progress: number } | { error: true }) => ({
...acc,
...curr
}),
{
file,
progress: 0,
error: false
}
)
)
)
)
),
mergeAll(3), // 3 upload in parallel maximum
scan(
(acc, curr) => ({
...acc,
// todo we can't use the File reference directly here
// but we shouldn't use the file name either
// instead we should generate a unique ID for each upload
[curr.file.name]: curr
}),
{}
),
map(fileEntities => Object.values(fileEntities))
);
public addFile() {
this.file$$.next(new File([], `test-file-${filesCount}`));
filesCount++;
}
}
Here's the code in stackblitz that you can fork: https://stackblitz.com/edit/rxjs-upload-multiple-files-v2?file=src/app/app.component.ts
I'm pretty close! If you open the live demo in stackblitz on the right and click on the "Add file" button, you'll see that you can add many files and they'll all get uploaded. The 3rd one will fail gracefully.
Now what is not working how I'd like:
If you click quickly more than 3 times on the "add file" button, only 3 files will appear in the queue. I'd like to have all of them but only 3 should be uploading at the same time. Yet, all the files to be uploaded should be displayed in the view, just waiting to start
The stop button should remove any upload. Whether it's uploading or failed
Thanks for any help
Number 1:
If you click quickly more than 3 times on the "add file" button, only 3 files will appear in the queue. I'd like to have all of them but only 3 should be uploading at the same time. Yet, all the files to be uploaded should be displayed in the view, just waiting to start
First of all, this is a cool problem because as far as I could see, you can't simply compose the existing operators (Without getting stupid with partition). You need a custom operator that splits your stream. If you don't want to subscribe to your source twice, you should share before splitting.
There's quite a lot of work left to implement your solution the way you'd like. BUT, in terms of getting your stream to show all files regardless of whether they're currently loading, there's really just one piece missing.
You want to split your stream. One stream should emit default
{
file,
progress: 0,
error: false
}`
files right away and the second stream should emit updates to those files. The second stream will have mergeAll(3), but the first doesn't need this limitation as it's not making a network request. You merge these two-streams and either update or add new entries into your output as you see fit.
Here's an example of that at work. I made a dummy example to abstract away the implementation details a bit. I start out with an array of objects with this shape,
{
id: number,
message: "HeyThere" + id,
response: none
}
I make a fake httpRequest call that enriches an object to
{
id: number,
message: "HeyThere" + id,
response: "Hello"
}
The stream emits each time a new object is added or when an object is enriched. But the enriching stream is limited to max 3 httpRequest calls at once.
const httpRequest= () => {
return timer(4000).pipe(
map(_ => "Hello")
);
}
const arrayO = [];
arrayO.length = 10;
from(arrayO).pipe(
map((val, index) => ({
id: index,
message: "HeyThere" + index,
response: "None"
})),
share(),
s => merge(s, s.pipe(
map(ob => httpRequest().pipe(
map(val => ({...ob, response: val}))
)),
mergeAll(3)
)),
scan((acc, val: any) => {
acc.set(val.id, val);
return acc;
}, new Map<number, any>()),
debounceTime(250),
map(mapO => Array.from(mapO.values()))
).subscribe(console.log);
I added a debounce as I find it makes the output much easier to follow. Since I added all 10 un-enriched objects synchronously, it just spams 10 arrays to the output if I don't debounce. Also, since every fake HttpRequest takes exactly 4 seconds, I get three arrays spammed at the output every 4 seconds. Debounce stops the UI from stuttering or the console from getting spammed.
Number 2
The stop button should remove any upload. Whether it's uploading or failed
This is a can of worms because every canonical solution says you should make a state management system. That would be the easiest way to interact with files that are in Queue, Loading, Failed, and Loaded all in one uniform way.
It's pretty easy to implement a lightweight Redux-style state management system using RxJS (Just use scan to manage state and JSON objects representing events to transform state). The toughest part is managing your current httpRequests. You'd probably create a custom mergeAll() operator that takes in events, removes queued requests, and even cancels mid-flight requests if necessary.
Using a stopFile$$ works to cancel mid-flight requests but it'll fall apart if people want to stop a fileload that hasn't started yet (as per your first requirement, you want those vsible too). It's sort of brittle regardless because emiting on a suject never comes with the assurance that anybody is listening. Another reason that a redux-style management is the way to go.
This is a very interesting problem, here is my approach to it:
uploadFile$ = this.uploadFile.pipe(
multicast(new Subject<CustomFile>(), subject =>
merge(
subject.pipe(
mergeMap(
// `file.id` might be created with uuid() or something like that
(file, idx) =>
of({ status: FILE_STATUS.PENDING, ...file }).pipe(
observeOn(asyncScheduler),
takeUntil(subject)
)
)
),
subject.pipe(
mergeMap(
(file, idx) =>
fakeUpload(file).pipe(
map(progress => ({
...file,
progress,
status: FILE_STATUS.LOADING
})),
startWith({
name: file.name,
status: FILE_STATUS.LOADING,
id: file.id,
progress: 0
}),
catchError(() => of({ ...file, status: FILE_STATUS.FAILED })),
scan(
(acc, curr) => ({
...acc,
...curr
}),
{} as CustomFile
),
takeUntil(
this.stopFile.pipe(
tap(console.warn),
filter(f => f.id === file.id)
)
)
),
3
)
)
)
)
);
files$: Observable<CustomFile[]> = merge(
this.uploadFile$,
this.stopFile
).pipe(
tap(v =>
v.status === FILE_STATUS.REMOVED ? console.warn(v) : console.log(v)
),
scan((filesAcc, crtFile) => {
// if the file is being removed, we need to remove it from the list
if (crtFile.status === FILE_STATUS.REMOVED) {
const { [crtFile.id]: _, ...rest } = filesAcc;
return rest;
}
// simply return an updated copy of the object when the file has the status either
// * `pending`(the buffer's length is > 3)
// * `loading`(the file is being uploaded)
// * `failed`(an error occurred during the file upload, but we keep it in the list)
// * `retrying`(the `Retry` button has been pressed)
return {
...filesAcc,
[crtFile.id]: crtFile
};
}, Object.create(null)),
// Might want to replace this by making the `scan`'s seed return an object that implements a custom iterator
map(obj => Object.values(obj))
);
StackBlitz demo.
I think the biggest problem here was how to determine when the mergeMap's buffer is full, so that a pending item should be shown to the user. As you can see, I've solved this using the multicast's second parameter:
multicast(new Subject(), subject => ...)
multicast(new Subject), refCount(), without its second argument, it's the same as share(). But when you provide the second argument(a.k.a the selector), you can achieve some sort of local multicasting:
if (isFunction(selector)) {
return operate((source, subscriber) => {
// the first argument
const subject = subjectFactory();
/* .... */
selector(subject).subscribe(subscriber).add(source.subscribe(subject));
});
}
selector(subject).subscribe(subscriber) will subscribe to the observable(which can also be a Subject) returned from the selector. Then, with .add(source.subscribe(subject)), the source is subscribed to. In the selector, we've used merge(subject.pipe(...), subject.pipe(...)), each of which will gain access to what's being pushed into the stream. Because of add(source.subscribe(subject)), the source's value will be passed to the Subject instance, which has its subscribers.
So, the way I solved the aforementioned problem was to create a race between observables. The first contender is
// #1
subject.pipe(
mergeMap(
// `file.id` might be created with uuid() or something like that
(file, idx) =>
of({ status: FILE_STATUS.PENDING, ...file }).pipe(
observeOn(asyncScheduler),
takeUntil(subject)
)
)
),
and the second one is
// #2
subject.pipe(
mergeMap(
(file, dx) => fileUpload().pipe(
/* ... */
// emits synchronously - as soon as the inner subscriber is created
startWith(...)
)
)
)
So, as soon as the Subject(the subject variable in this case) receives the value from the source, it will send it to all of its subscribers - the 2 contenders. It all happens synchronously, which also means that the order matters. #1 will be the first subscriber to receive the value, and #2 will be second. The way the winner is selected is to see which one of the 2 subscribers emits first.
Notice that the first will pass along the value asynchronously(with the help of observeOn(asyncScheduler)) and the second one synchronously. The first one will emit first if the buffer is full, otherwise the second will emit.
I've ended up with the following:
export interface FileUpload {
file: File;
progress: number;
error: boolean;
toRemove: boolean;
}
export const uploadManager = () => {
const file$$: Subject<File> = new Subject();
const retryFile$$: Subject<File> = new Subject();
const stopFile$$: Subject<File> = new Subject();
const fileStartOrRetry$: Observable<File> = file$$.pipe(
mergeMap(file =>
retryFile$$.pipe(
filter(retryFile => retryFile === file),
startWith(file)
)
),
share()
);
const addFileToQueueAfterStartOrRetry$: Observable<
FileUpload
> = fileStartOrRetry$.pipe(
map(file => ({
file,
progress: 0,
error: false,
toRemove: false
}))
);
const markFileToBeRemovedAfterStop$: Observable<FileUpload> = stopFile$$.pipe(
map(file => ({
file,
progress: 0,
error: false,
toRemove: true
}))
);
const updateFileProgress$: Observable<FileUpload> = fileStartOrRetry$.pipe(
map(file =>
uploadMock().pipe(
map(progress => ({ progress })),
takeUntil(
stopFile$$.pipe(filter(stopFile => stopFile.name === file.name))
),
catchError(() => of({ error: true })),
scan(
(acc, curr: { progress: number } | { error: true }) => ({
...acc,
...curr
}),
{
file,
progress: 0,
error: false,
toRemove: false
}
)
)
),
// 3 upload in parallel maximum
mergeAll(3)
);
const files$: Observable<FileUpload[]> = merge(
addFileToQueueAfterStartOrRetry$,
updateFileProgress$,
markFileToBeRemovedAfterStop$
).pipe(
scan<FileUpload, { [key: string]: FileUpload }>((acc, curr) => {
if (curr.toRemove) {
const copy = { ...acc };
delete copy[curr.file.name];
return copy;
}
return {
...acc,
// todo we can't use the File reference directly here
// but we shouldn't use the file name either
// instead we should generate a unique ID for each upload
[curr.file.name]: curr
};
}, {}),
map(fileEntities => Object.values(fileEntities))
);
return {
files$,
file$$,
retryFile$$,
stopFile$$
};
};
It covers all the cases as demonstrated here: https://rxjs-upload-multiple-file-v3.stackblitz.io
The code is here: https://stackblitz.com/edit/rxjs-upload-multiple-file-v3?file=src/app/upload-manager.ts
It's based on Mrk Sef's suggestion. It clicked after he mentioned "You want to split your stream".
I have some javascript:
this.mySubscription = someObservable.subscribe((obs: any) => {
this.mySubscription.unsubscribe();
this.mySubscription = undefined;
}
on execution, the console logs the error ERROR TypeError: Cannot read property 'unsubscribe' of undefined.
I wonder why I can not unsubscribe inside the subscribe lambda function. Is there a correct way to do so? I have read a bit about using dummy-subjects and completing them or using takeUntil/takeWhile and other pipe operators workArounds.
What is a correct way/workaround to unsubscribe a subscription inside the subscription's subscribe-function?
I am currently using a dummy subscription like so:
mySubscription: BehaviorSubject<any> = new BehaviorSubject<any>(undefined);
// when I do the subscription:
dummySubscription: BehaviorSubject<any> = new BehaviourSubject<any>(this.mySubscription.getValue());
this.mySubscription = someObservable.subscribe((obs: any) => {
// any work...
dummySubscription.next(obs);
dummySubscription.complete();
dummySubscription = undefined;
}, error => {
dummySubscription.error(error);
});
dummySubscription.subscribe((obs: any) => {
// here the actual work to do when mySubscription emits a value, before it should have been unsubscribed upon
}, err => {
// if errors need be
});
You shouldn't try to unsubscribe in the subscribe function.
You can unsubscribe with operators like take, takeWhile or takeUntil.
take
Use take(n) to unsubscribe after someObservable emits n times.
someObservable.pipe(
take(1)
).subscribe(value => console.log(value));
takeWhile
Use takeWhile to unsubscribe when an emitted value fails a condition.
someObservable.pipe(
takeWhile(value => valueIsSave(value))
).subscribe(value => console.log(value));
valueIsSave(value): boolean {
// return true if the subscription should continue
// return false if you want to unsubscribe on that value
}
takeUntil
Use takeUntil(obs$) to unsubscribe when the observable obs$ emits.
const terminate = new Subject();
someObservable.pipe(
takeUntil(terminate)
).subscribe(value => console.log(value));
unsub() {
terminate.next() // trigger unsubscribe
}
If you make your stream asynchronous, what you're doing should work. For example, this will not work:
const sub = from([1,2,3,4,5,6,7,8,9,10]).subscribe(val => {
console.log(val);
if(val > 5) sub.unsubscribe();
});
but this will work:
const sub2 = from([1,2,3,4,5,6,7,8,9,10]).pipe(
delay(0)
).subscribe(val => {
console.log(val);
if(val > 5) sub2.unsubscribe();
});
Because the JS event loop is fairly predictable (blocks of code are always run to completion), If any part of your stream is asynchronous, then you can be sure that your subscription will be defined before your lambda callback is invoked.
Should you do this?
Probably not. If your code relies on the internal (otherwise hidden) machinations of your language/compiler/interpreter/etc, you've created brittle code and/or code that is hard to maintain. The next developer looking at my code is going to be confused as to why there's a delay(0) - that looks like it shouldn't do anything.
Notice that in subscribe(), your lambda has access to its closure as well as the current stream variable. The takeWhile() operator has access to the same closure and the same stream variables.
from([1,2,3,4,5,6,7,8,9,10]).pipe(
takeWhile(val => {
// add custom logic
return val <= 5;
})
).subscribe(val => {
console.log(val);
});
takeWhile() can to anything that sub = subscribe(... sub.unsubscibe() ... ), and has the added benefit of not requiring you to manage a subscription object and being easier to read/maintain.
Inspired by another answer here and especially this article, https://medium.com/#benlesh/rxjs-dont-unsubscribe-6753ed4fda87, I'd like to suggest takeUntil() with following example:
...
let stop$: Subject<any> = new Subject<any>(); // This is the one which will stop the observable ( unsubscribe a like mechanism )
obs$
.pipe(
takeUntil(stop$)
)
.subscribe(res => {
if ( res.something === true ) {
// This next to lines will cause the subscribe to stop
stop$.next();
stop$.complete();
}
});
...
And I'd like to quote sentence RxJS: Don’t Unsubscribe from those article title mentioned above :).
I'm using jest to test a redux-observable epic that forks off an inner observable created using Observable.fromEvent and listens for a specific keypress before emitting an action.
I'm struggling to test for when the inner Observable does not receive this specific keypress and therefore does not emit an action.
Using jest, the following times out:
import { Observable, Subject } from 'rxjs'
import { ActionsObservable } from 'redux-observable'
import keycode from 'keycode'
const closeOnEscKeyEpic = action$ =>
action$.ofType('LISTEN_FOR_ESC').switchMapTo(
Observable.fromEvent(document, 'keyup')
.first(event => keycode(event) === 'esc')
.mapTo({ type: 'ESC_PRESSED' })
)
const testEpic = ({ setup, test, expect }) =>
new Promise(resolve => {
const input$ = new Subject()
setup(new ActionsObservable(input$))
.toArray()
.subscribe(resolve)
test(input$)
}).then(expect)
// This times out
it('no action emitted if esc key is not pressed', () => {
expect.assertions(1)
return testEpic({
setup: input$ => closeOnEscKeyEpic(input$),
test: input$ => {
// start listening
input$.next({ type: 'LISTEN_FOR_ESC' })
// press the wrong keys
const event = new KeyboardEvent('keyup', {
keyCode: keycode('p'),
})
const event2 = new KeyboardEvent('keyup', {
keyCode: keycode('1'),
})
global.document.dispatchEvent(event)
global.document.dispatchEvent(event2)
// end test
input$.complete()
},
expect: actions => {
expect(actions).toEqual([])
},
})
})
My expectation was that calling input$.complete() would cause the promise in testEpic to resolve, but for this test it does not.
I feel like I'm missing something. Does anyone understand why this is not working?
I'm still new to Rx/RxJS, so my apologies if the terminology of this answer is off. I was able to reproduce your scenario, though.
The inner observable (Observable.fromEvent) is blocking the outer observable. The completed event on your ActionsObservable doesn't propagate through until after the inner observable is completed.
Try out the following code snippet with this test script:
Run the code snippet.
Press a non-Escape key.
Nothing should be printed to the console.
Select the "Listen for Escape!" button.
Press a non-Escape key.
The keyCode should be printed to the console.
Select the "Complete!" button.
Press a non-Escape key.
The keyCode should be printed to the console.
Press the Escape key.
The keyCode should be printed to the console
The onNext callback should print the ESC_PRESSED action to the console.
The onComplete callback should print to the console.
document.getElementById('complete').onclick = onComplete
document.getElementById('listenForEsc').onclick = onListenForEsc
const actions = new Rx.Subject()
const epic = action$ =>
action$.pipe(
Rx.operators.filter(action => action.type === 'LISTEN_FOR_ESC'),
Rx.operators.switchMapTo(
Rx.Observable.fromEvent(document, 'keyup').pipe(
Rx.operators.tap(event => { console.log('keyup: %s', event.keyCode) }),
Rx.operators.first(event => event.keyCode === 27), // escape
Rx.operators.mapTo({ type: 'ESC_PRESSED' }),
)
)
)
epic(actions.asObservable()).subscribe(
action => { console.log('next: %O', action) },
error => { console.log('error: %O', error) },
() => { console.log('complete') },
)
function onListenForEsc() {
actions.next({ type: 'LISTEN_FOR_ESC' })
}
function onComplete() {
actions.complete()
}
<script src="https://unpkg.com/rxjs#5.5.0/bundles/Rx.min.js"></script>
<button id="complete">Complete!</button>
<button id="listenForEsc">Listen for Escape!</button>
Neither the switchMapTo marble diagram nor its textual documentation) clearly indicate what happens when the source observable completes before the inner observable. However, the above code snippet demonstrates exactly what you observed in the Jest test.
I believe this answers your "why" question, but I'm not sure I have a clear solution for you. One option could be to hook in a cancellation action and use takeUntil on the inner observable. But, that might feel awkward if that's only ever used in your Jest test.
I can see how this epic/pattern wouldn't be a problem in a real application as, commonly, epics are created and subscribed to once without ever being unsubscribed from. However, depending on the specific scenario (e.g. creating/destroying the store multiple times in a single application), I could see this leading to hung subscriptions and potential memory leaks. Good to keep in mind!