I am trying to achieve the following with Rxjs: given an array of job ids, for every id in the array, poll an endpoint that returns the status of the job. The status can be either "RUNNING", or "FINISHED". The code should poll jobs one after the other, and continue the polling until the jobs are in the "RUNNING" status. As soon as a job reaches the "FINISHED" status, it should be passed downstream, and excluded from further polling.
Below is a minimal toy case that demonstrates the problem.
const {
from,
of,
interval,
mergeMap,
filter,
take,
tap,
delay
} = rxjs;
const { range } = _;
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input })
.pipe(delay(500));
};
const number$ = from(range(1, 10));
const poll = (number) => interval(5000).pipe(
mergeMap(() => {
return doRequest(number)
}),
tap(console.log),
filter(( {status} ) => status === 'FINISHED'),
take(1)
);
const printout$ = number$.pipe(
mergeMap((number) => {
return poll(number)
})
);
printout$.subscribe(console.log);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.21/lodash.min.js"></script>
It does most of what I described; but it polls all endpoints simultaneously rather than one after another. Here, roughly, is the pattern I would like to achieve:
starting with ids: [1, 2, 3]
polling: await request 1 then await request 2 then await request 3
then wait for n seconds; then repeat
after job 2 is finished, send request 1, then send request 3, then wait, then repeat
after job 3 is finished, send request 1, then wait, repeat
after job 1 is finished, complete the stream
I feel that in order to achieve the sending of the requests in sequence, they should be concatMaped; but in the snippet above that's not possible because of the interval that would prevent each polling stream from terminating.
Could you please advise how to modify my code to achieve what I am describing?
If I understand the problem right, I would proceed like this.
First of all I would create a poll function that returns an Observable which notifies after a round of pollings, and it emits an array of all numbers for which the call to doRequest returns 'RUNNING'. Such a function would look something like this
const poll = (numbers: number[]) => {
return from(numbers).pipe(
concatMap((n) =>
doRequest(n).pipe(
filter((resp) => resp.status === 'RUNNING'),
map((resp) => resp.value)
)
),
toArray()
);
};
Then what you need to do is to recursively iterate a call the poll function until the array emitted by the Observable returned by poll is empty.
Recursion in rxjs is obtained typically with the expand operator, and this is the operator which we are going to use also in this case, like this
poll(numbers)
.pipe(
expand((numbers) =>
numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)))
)
)
.subscribe(console.log);
A complete example can be seen in this stackblitz.
UPDATE
If the objective is to notify the job ids which have finished with a polling logic, the structure of the solution remains the same (a poll function and recursivity via expand) but the details are different.
The poll function makes sure we emit all the responses of a polling round and it looks like this:
const poll = (
numbers: number[]
) => {
console.log(`Polling ${numbers}`);
return from(numbers).pipe(
concatMap((n) => doRequest(n)),
toArray()
);
};
The recursion logic makes sure that all jobs that are still with "RUNNING" status are polled again but then we filter only the jobs which are FINISHED and passed them downstream. In other words the logic looks like this
poll(start)
.pipe(
expand((responses) => {
const numbers = responses.filter(r => r.status === 'RUNNING').map(r => r.value)
return numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)));
}),
map(responses => responses.filter(r => r.status === 'FINISHED')),
filter(finished => finished.length > 0)
)
.subscribe({
next: responses => console.log(`Job finished ${responses.map(r => r.value)}`),
complete: () => {console.log('All processed')}
});
A working example can be seen in this stackblitz.
Updated: Original answer was not on the right track.
What we want to achieve is that on each go around of the interval we poll all the outstanding jobs in order. We yield up any completed jobs to the output observable and we also omit those completed jobs from subsequent polls.
We can do that by using a Subject instead of a static observable of the job IDs. We start our poll interval and we use withLatestFrom to include the latest list of job IDs. We can then add a tap into the output observable when we get a finished job and update the Subject to omit that job.
To end the poller interval we can create an observable that fires when the array of outstanding jobs is empty and use takeUntil with that.
const number$ = new Subject();
const noMoreNumber$ = number$.pipe(skipWhile((numbers) => numbers.length > 0));
const printout$ = interval(5000).pipe(
withLatestFrom(number$),
switchMap(([_, numbers]) => {
return numbers.map((number) => defer(() => doRequest(number)));
}),
concatAll(),
//tap(console.log),
filter(({ status }) => status === 'FINISHED'),
withLatestFrom(number$),
tap(([{ value }, numbers]) =>
number$.next(numbers.filter((num) => num != value))
),
map(([item]) => item),
takeUntil(noMoreNumber$)
);
printout$.subscribe({
next: console.log,
error: console.error,
complete: () => console.log('COMPLETE'),
});
number$.next([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
The other tweak I would make is to use switchMap instead of mergeMap inside the poller itself. If you use that in combination with fromFetch for performing your HTTP calls then, if there is some long-running HTTP call which gets stuck, on the next poll the previous call will be cancelled before it makes the next HTTP call because switchMap disposes of the previous observable before subscribing to the new one.
Here's a working example:
https://stackblitz.com/edit/js-gxrrb3?devToolsHeight=33&file=index.js
Generates console output looking like this...
TRY this
import { delay, EMPTY, from, of, range } from 'rxjs';
import { concatMap, filter, mergeMap, tap, toArray } from 'rxjs/operators';
const number$ = from(range(1, 3));
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input }).pipe(delay(1000));
};
const poll = (jobs: object[]) => {
return from(jobs).pipe(
filter((job) => job['status'] !== 'FINISHED'),
concatMap((job) => doRequest(job['value'])),
tap((job) => {
console.log('polling with................', job);
}),
toArray(),
tap((result) => {
console.log('curent jobs................', JSON.stringify(result));
}),
mergeMap((result) =>
result.length > 0 ? poll(result) : of('All job completed!')
)
);
};
const initiateJob = number$.pipe(
mergeMap((id) => doRequest(id)),
toArray(),
tap((jobs) => {
console.log('initialJobs: ', JSON.stringify(jobs));
}),
concatMap(poll)
);
initiateJob.subscribe({
next: console.log,
error: console.log,
complete: () => console.log('COMPLETED'),
});
After reading a large number of posts on stack overflow, I am still very confused about the difference between Observable and Subject in Rx.
Basically, most people point out that the key difference between the two is that "The Subject class inherits both Observable and Observer, in the sense that it is both an observer and an observable", and it can be demonstrated by the following code snippet:
var subject = new Rx.Subject();
var subscription = subject.subscribe(
function (x) { console.log('onNext: ' + x); },
function (e) { console.log('onError: ' + e.message); },
function () { console.log('onCompleted'); }
);
subject.onNext(1);
// => onNext: 1
subject.onNext(2);
// => onNext: 2
subject.onCompleted();
// => onCompleted
I understand that in the following part of the code, subject is being the role of an observable that can be subscribed to:
var subscription = subject.subscribe(
function (x) { console.log('onNext: ' + x); },
function (e) { console.log('onError: ' + e.message); },
function () { console.log('onCompleted'); }
);
What really confuses me is this part:
subject.onNext(1);
// => onNext: 1
subject.onNext(2);
// => onNext: 2
subject.onCompleted();
// => onCompleted
It looks like here it demonstrates that subject can be the role of an observer as well, but my understanding is that here subject is also feeding values/events (sending 1, 2 and complete event). So why is the subject considered the role of an observer in the code above?
Thanks a lot.
The subject is considered as taking the role of an observer in the referenced code precisely because onNext and onCompleted are being called on it. An Observer is defined as an object on which you can call onNext and onCompleted (as well as onError.)
I am using socket.io to send a series of responses to my front-end. The responses are intended to be sequential, but depending on the connection created by socket.io they're not always guaranteed to come in the correct order (https://github.com/josephg/ShareJS/issues/375).
Assuming each response had a sequence field that held a number (shown as the number in the picture above), the observable should emit these responses in order.
If a response is received out of order and a certain amount of time (n) passes without getting any response, I would like for my observable to emit an error, to signal to my front-end to reset the connection.
A really nice problem. Below a snippet with most important parts commented.
// mock ordered values
const mockMessages = Rx.Observable.fromEvent(document.querySelector('#emit'), 'click')
.map((e, index) => ({
index,
timestamp: e.timeStamp
}))
.delayWhen(() => Rx.Observable.timer(Math.random() * 2000)) // distort order
// there is a lot of mutability in `keepOrder`, but all of it
// is sealed and does not leak to outside environment
const keepOrder = timeoutMs => stream =>
Rx.Observable.defer(() => // need defer to support retries on error
stream.scan((acc, v) => {
acc.buffer.push(v)
acc.buffer.sort((v1, v2) => v1.index - v2.index)
return acc
}, {
lastEmitted: -1,
buffer: []
})
.mergeMap(info => {
const emission = []
while (info.buffer.length && info.lastEmitted + 1 === info.buffer[0].index) {
emission.push(info.buffer.shift())
info.lastEmitted += 1
}
return Rx.Observable.of(emission)
})
.switchMap(emissions => {
if (!emissions.length) { // this condition indicates out of order
return Rx.Observable.timer(timeoutMs)
.mergeMapTo(Rx.Observable
.throw(new Error('ORDER_TIMEOUT')))
} else {
return Rx.Observable.from(emissions)
}
})
)
mockMessages
.do(x => console.log('mocked', x.index))
.let(keepOrder(1000)) // decrease timeoutMs to increase error probablity
.do(x => console.log('ORDERED', x.index))
.retryWhen(es => es
.do(e => console.warn('ERROR', e)))
.subscribe()
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.2/Rx.js"></script>
<button id="emit">EMIT</button>
Consider the following code:
this.msgService.getUserChatList()
.do( (list) => {
this.unread = false;
console.log(list);
} )
.mergeMap( chats => Observable.from(chats) )
.mergeMap( chat => this.msgService.getLastMessage(chat['id']).map( lastMessage => this.containsUnreadMessages(lastMessage, chat['lastPresence']) ) )
.filter( state => state === true )
.subscribe( (unread) => {
this.unread = true;
console.log('result ', res);
} );
getUserChatList():
- emits an element each time one of the chat changes
- an element is a raw array containing all chats meta-data
- never completes
getLastMessage():
- is an Observable that never completes
In the second mergeMap I am calling the function getLastMessage().
I need to be observe this observable only until a new item is emitted by the getUserChatList() otherwise I would multiple observations on last message of the same chat.
Illustration :
getUserChatList emits : [chatMetaA:{}, chatMetaB:{}]
code go through getLastMessage and start to observe lastMessage of chatA and chatB
one of the chat change so a new item is emitted by getUserChatList containing the new version of the meta-data of the chats: [chatMetaA:{}, chatMetaB:{}]
code go through getLastMessage and start to observe lastMessage of chatA and chatB. So we now observe twice last message of chatA and chatB
And it will go on and on...
My question is, how could I cancel observation on getLastMessage() once a new item is emitted by getUserChatList()? I tried using switch but couldn't manage to make it work
Solution was indeed to use switchMap:
this.msgService.getUserChatList()
.do( () => { this.unread = false } )
.switchMap(
chats => Observable.from(chats)
.mergeMap( chat => this.msgService.getLastMessage(chat['id'])
.map( lastMessage => this.containsUnreadMessages(lastMessage, chat['lastPresence']) ) )
)
.filter( state => state === true )
.subscribe( (unread) => {
this.unread = true;
console.log('result ', res);
} );
I'm building out an Angular2 app, and have two BehaviourSubjects that I want to logically combine into one subscription. I'm making two http requests and want to fire an event when both of them come back. I'm looking at forkJoin vs combineLatest. It seems that combineLatest will fire when either behvaviorSubjects are updated vs forkJoin will fire only after all behavoirSubjects are updated. Is this correct? There has to be a generally accepted pattern for this isn't there?
EDIT
Here is an example of one of my behaviorSubjects my angular2 component is subscribing to:
export class CpmService {
public cpmSubject: BehaviorSubject<Cpm[]>;
constructor(private _http: Http) {
this.cpmSubject = new BehaviorSubject<Cpm[]>(new Array<Cpm>());
}
getCpm(id: number): void {
let params: URLSearchParams = new URLSearchParams();
params.set('Id', id.toString());
this._http.get('a/Url/Here', { search: params })
.map(response => <Cpm>response.json())
.subscribe(_cpm => {
this.cpmSubject.subscribe(cpmList => {
//double check we dont already have the cpm in the observable, if we dont have it, push it and call next to propigate new cpmlist everywheres
if (! (cpmList.filter((cpm: Cpm) => cpm.id === _cpm.id).length > 0) ) {
cpmList.push(_cpm);
this.cpmSubject.next(cpmList);
}
})
});
}
}
Here is a snippet of my component's subscription:
this._cpmService.cpmSubject.subscribe(cpmList => {
doSomeWork();
});
But instead of firing doSomeWork() on the single subscription I want to only fire doSomeWork() when the cpmSubject and fooSubject fire.
You could use the zip-operator, which works similar to combineLatest or forkJoin, but triggers only when both streams have emitted: http://reactivex.io/documentation/operators/zip.html
The difference between zip and combineLatest is:
Zip will only trigger "in parallel", whereas combineLatest will trigger with any update and emit the latest value of each stream.
So, assuming the following 2 streams:
streamA => 1--2--3
streamB => 10-20-30
with zip:
"1, 10"
"2, 20"
"3, 30"
with combineLatest:
"1, 10"
"2, 10"
"2, 20"
"3, 20"
"3, 30"
Here is also a live-example:
const a = new Rx.Subject();
const b = new Rx.Subject();
Rx.Observable.zip(a,b)
.subscribe(x => console.log("zip: " + x.join(", ")));
Rx.Observable.combineLatest(a,b)
.subscribe(x => console.log("combineLatest: " + x.join(", ")));
a.next(1);
b.next(10);
a.next(2);
b.next(20);
a.next(3);
b.next(30);
<script src="https://unpkg.com/rxjs/bundles/Rx.min.js"></script>
Also another sidenote: Never ever ever subscribe inside a subscribe.
Do something like this instead:
this._http.get('a/Url/Here', { search: params })
.map(response => <Cpm>response.json())
.withLatestFrom(this.cpmSubject)
.subscribe([_cpm, cpmList] => {
if (! (cpmList.filter((cpm: Cpm) => cpm.id === _cpm.id).length > 0) ) {
cpmList.push(_cpm);
this.cpmSubject.next(cpmList);
}
});