I am trying to achieve the following with Rxjs: given an array of job ids, for every id in the array, poll an endpoint that returns the status of the job. The status can be either "RUNNING", or "FINISHED". The code should poll jobs one after the other, and continue the polling until the jobs are in the "RUNNING" status. As soon as a job reaches the "FINISHED" status, it should be passed downstream, and excluded from further polling.
Below is a minimal toy case that demonstrates the problem.
const {
from,
of,
interval,
mergeMap,
filter,
take,
tap,
delay
} = rxjs;
const { range } = _;
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input })
.pipe(delay(500));
};
const number$ = from(range(1, 10));
const poll = (number) => interval(5000).pipe(
mergeMap(() => {
return doRequest(number)
}),
tap(console.log),
filter(( {status} ) => status === 'FINISHED'),
take(1)
);
const printout$ = number$.pipe(
mergeMap((number) => {
return poll(number)
})
);
printout$.subscribe(console.log);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.21/lodash.min.js"></script>
It does most of what I described; but it polls all endpoints simultaneously rather than one after another. Here, roughly, is the pattern I would like to achieve:
starting with ids: [1, 2, 3]
polling: await request 1 then await request 2 then await request 3
then wait for n seconds; then repeat
after job 2 is finished, send request 1, then send request 3, then wait, then repeat
after job 3 is finished, send request 1, then wait, repeat
after job 1 is finished, complete the stream
I feel that in order to achieve the sending of the requests in sequence, they should be concatMaped; but in the snippet above that's not possible because of the interval that would prevent each polling stream from terminating.
Could you please advise how to modify my code to achieve what I am describing?
If I understand the problem right, I would proceed like this.
First of all I would create a poll function that returns an Observable which notifies after a round of pollings, and it emits an array of all numbers for which the call to doRequest returns 'RUNNING'. Such a function would look something like this
const poll = (numbers: number[]) => {
return from(numbers).pipe(
concatMap((n) =>
doRequest(n).pipe(
filter((resp) => resp.status === 'RUNNING'),
map((resp) => resp.value)
)
),
toArray()
);
};
Then what you need to do is to recursively iterate a call the poll function until the array emitted by the Observable returned by poll is empty.
Recursion in rxjs is obtained typically with the expand operator, and this is the operator which we are going to use also in this case, like this
poll(numbers)
.pipe(
expand((numbers) =>
numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)))
)
)
.subscribe(console.log);
A complete example can be seen in this stackblitz.
UPDATE
If the objective is to notify the job ids which have finished with a polling logic, the structure of the solution remains the same (a poll function and recursivity via expand) but the details are different.
The poll function makes sure we emit all the responses of a polling round and it looks like this:
const poll = (
numbers: number[]
) => {
console.log(`Polling ${numbers}`);
return from(numbers).pipe(
concatMap((n) => doRequest(n)),
toArray()
);
};
The recursion logic makes sure that all jobs that are still with "RUNNING" status are polled again but then we filter only the jobs which are FINISHED and passed them downstream. In other words the logic looks like this
poll(start)
.pipe(
expand((responses) => {
const numbers = responses.filter(r => r.status === 'RUNNING').map(r => r.value)
return numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)));
}),
map(responses => responses.filter(r => r.status === 'FINISHED')),
filter(finished => finished.length > 0)
)
.subscribe({
next: responses => console.log(`Job finished ${responses.map(r => r.value)}`),
complete: () => {console.log('All processed')}
});
A working example can be seen in this stackblitz.
Updated: Original answer was not on the right track.
What we want to achieve is that on each go around of the interval we poll all the outstanding jobs in order. We yield up any completed jobs to the output observable and we also omit those completed jobs from subsequent polls.
We can do that by using a Subject instead of a static observable of the job IDs. We start our poll interval and we use withLatestFrom to include the latest list of job IDs. We can then add a tap into the output observable when we get a finished job and update the Subject to omit that job.
To end the poller interval we can create an observable that fires when the array of outstanding jobs is empty and use takeUntil with that.
const number$ = new Subject();
const noMoreNumber$ = number$.pipe(skipWhile((numbers) => numbers.length > 0));
const printout$ = interval(5000).pipe(
withLatestFrom(number$),
switchMap(([_, numbers]) => {
return numbers.map((number) => defer(() => doRequest(number)));
}),
concatAll(),
//tap(console.log),
filter(({ status }) => status === 'FINISHED'),
withLatestFrom(number$),
tap(([{ value }, numbers]) =>
number$.next(numbers.filter((num) => num != value))
),
map(([item]) => item),
takeUntil(noMoreNumber$)
);
printout$.subscribe({
next: console.log,
error: console.error,
complete: () => console.log('COMPLETE'),
});
number$.next([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
The other tweak I would make is to use switchMap instead of mergeMap inside the poller itself. If you use that in combination with fromFetch for performing your HTTP calls then, if there is some long-running HTTP call which gets stuck, on the next poll the previous call will be cancelled before it makes the next HTTP call because switchMap disposes of the previous observable before subscribing to the new one.
Here's a working example:
https://stackblitz.com/edit/js-gxrrb3?devToolsHeight=33&file=index.js
Generates console output looking like this...
TRY this
import { delay, EMPTY, from, of, range } from 'rxjs';
import { concatMap, filter, mergeMap, tap, toArray } from 'rxjs/operators';
const number$ = from(range(1, 3));
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input }).pipe(delay(1000));
};
const poll = (jobs: object[]) => {
return from(jobs).pipe(
filter((job) => job['status'] !== 'FINISHED'),
concatMap((job) => doRequest(job['value'])),
tap((job) => {
console.log('polling with................', job);
}),
toArray(),
tap((result) => {
console.log('curent jobs................', JSON.stringify(result));
}),
mergeMap((result) =>
result.length > 0 ? poll(result) : of('All job completed!')
)
);
};
const initiateJob = number$.pipe(
mergeMap((id) => doRequest(id)),
toArray(),
tap((jobs) => {
console.log('initialJobs: ', JSON.stringify(jobs));
}),
concatMap(poll)
);
initiateJob.subscribe({
next: console.log,
error: console.log,
complete: () => console.log('COMPLETED'),
});
I obtain my Ofertas here
getOfertasByYear(year:number): Observable<Oferta[]> {
return this.http.get<Oferta[]>(`${this.urlWebAPI}/ofertas/year/${year}`)
.pipe(
map(ofertas=>
ofertas.map(oferta=>({
...oferta,
añoPresentada:new Date(oferta.fechaPresentacionFulcrum).getFullYear(),
organismoId:¿¿¿???
}) as Oferta)
),
tap(data => console.log('OfertasService-getOfertasByYear(): ', data)
),
catchError(this.handleError)
)
}
But I need to calculate his organismoId and that is here
getOrganismoDeOferta(ofertaId:string): Observable<Organismo> {
return this.http.get<Organismo>(`${this.urlWebAPI}/organismos/oferta/${ofertaId}`)
.pipe(
tap(//data=>console.log('OfertasService-getOrganismos(): ', data)
),
catchError(this.handleError)
)
}
And I don't know how to pass the result of this Observable to te mapped property
getOfertasByYear(year:number): Observable<Oferta[]> {
return this.http.get<Oferta[]>(`${this.urlWebAPI}/ofertas/year/${year}`)
.pipe(
map(ofertas=>
ofertas.map(oferta=>({
...oferta,
añoPresentada:new Date(oferta.fechaPresentacionFulcrum).getFullYear(),
organismoId:this.getOrganismoDeOferta(oferta.id).subscribe(data=>{
¿¿¿¿??????
})
}) as Oferta)
),
tap(data => console.log('OfertasService-getOfertasByYear(): ', data)
),
catchError(this.handleError)
)
}
I subscribe to it but I don't know how to make the assignment
I have tried to obtain all Ofertas and All Concursos but neither
ofertas$ = this.dataService.getOfertas();
concursos$ = this.dataService.getConcursos();
ofertasConOrganismos$ = forkJoin([
this.ofertas$,
this.concursos$
])
.pipe(
map(([ofertas, concursos]) =>
ofertas.map(oferta => ({
...oferta,
organismoId: concursos.find(c => c.id == oferta.concursoId).organismoId
}) as Oferta))
);
But I get this error:
Cannot read properties of undefined (reading 'organismoId')
Any idea, please?
Thanks
Instead of using a plain map and then calling .subscribe(), you can use a "Higher-Order Mapping Operator" to handle the inner subscription for you. In this case, let's use switchMap.
The idea is to return an observable inside switchMap that emits the data you need. Since you need to make multiple calls, we can leverage some help from forkJoin.
With forkJoin you pass in an array of observables, and it will emit an array of the results. So here below we map the array of Oferta to an array of observables that will each emit the Oferta with the organismoId appended:
getOfertasByYear(year: number): Observable<Oferta[]> {
return this.http.get<Oferta[]>(`${this.urlWebAPI}/ofertas/year/${year}`).pipe(
switchMap(ofertas => forkJoin(
ofertas.map(oferta => this.appendOrganismo(oferta))
)),
catchError(this.handleError)
)
}
Nothing too fancy for the definition of appendOrganismo(); we just make the http call, then map the result to the desired shape:
private appendOrganismo(oferta: Oferta) {
return this.getOrganismoDeOferta(oferta.id).pipe(
map(organismo => ({
...oferta,
añoPresentada: new Date(oferta.fechaPresentacionFulcrum).getFullYear(),
organismoId: organismo.id
}))
);
}
I have to create a queue of ajax requests and group the result but i have no clue about to accomplish this.
Let's say i have an array like this:
const actors = [
"Al Pacino",
"Robert De Niro"
];
I have to iterate over it and for each values make an api call:
export const getMovies = action$ =>
action$.pipe(
ofType(LOAD_REQUEST),
// iterate over the array
// make api call with the actor name
// for each result, make a second api call with the id of the actor (get in the response.results.id)
// group the result in an array with all films of all actors of the array passed in the payload
);
Im stuck with switchMap, pipe ... and don't know the correct way to accomplish this.
Edit Tried your solution Valeriy but got this error:
export const getMovies = action$ =>
action$.pipe(
ofType(LOAD_REQUEST),
switchMap(({ payload }) =>
combineLatest(
payload.map(a => {
return ajax
.getJSON(actor(a))
.pipe(map(response => console.log(response)));
})
)
)
);
TypeError: You provided 'function (source) {
return source.lift.call(Object(_observable_from__WEBPACK_IMPORTED_MODULE_2__["from"])([source].concat(observables)), new _observable_combineLatest__WEBPACK_IMPORTED_MODULE_1__["CombineLatestOperator"](project));
}' where a stream was expected. You can provide an Observable, Promise, Array, or Iterable.
If I understand you correctly, you are trying to achieve something like this:
export const getMovies = action$ => action$.pipe(
ofType(LOAD_REQUEST),
switchMap(() => {
// group the result in an array with all films of all actors of the array passed in the payload
return combineLatest(
// iterate over the array
...actors.map(actorName => {
// make api call with the actor name
return loadActor(actorName).pipe(
// for each result, make a second api call with the id of the actor (get in the response.results.id)
switchMap(response => loadActorFilms(response.results.id))
);
})
);
})
);
I've used combineLatest to group multiple observables together.
I call backend that respond with:
[
"https://some-url.com/someData1.json",
"https://some-url.com/someData2.json"
]
Each JSON can have following schema:
{
"isValid": boolean,
"data": string
}
I want to get array with all data, that have isValid is set to true
backend.get(url)
.pipe(
mergeMap((urls: []) =>
urls.map((url: string) =>
backend.get(url)
.pipe(
filter(response => response.isValid),
map(response => response.data)
)
)
),
combineAll()
)
When both .json have "isValid" set to true, I get array with both data.
But when one of them has "isValid" set to false observable never completes.
I could use mergeAll instead of combineAll, but then I receive stream of single data not collection of all data.
Is there any better way to filter out observable?
As you said, the inner observable never emits, because filter does not forward the only value that is ever emitted by the backend.get observable. In that case, the operator subscribing on that observable - in your case combineAll - will also never receive any value and cannot ever emit itself.
What I would do is just move the filtering and mapping to combineAll by providing a project function, like that:
backend.get(url)
.pipe(
mergeMap((urls: string[]) =>
urls.map((url: string) => backend.get(url))
),
combineAll(responses =>
responses
.filter(response => response.isValid)
.map(response => response.data)
)
)
See if that works for you ;)
import { forkJoin, Observable } from 'rxjs';
import { map } from 'rxjs/operators';
interface IRes {
isValid: boolean;
data: string;
}
interface IResValid {
isValid: true;
data: string;
}
function isValid(data: IRes): data is IResValid {
return data.isValid;
}
const res1$: Observable<IRes> = backend.get(url1);
const res2$: Observable<IRes> = backend.get(url2);
// When all observables complete, emit the last emitted value from each.
forkJoin([res1$, res2$])
.pipe(map((results: IRes[]) => results.filter(isValid)))
.subscribe((results: IResValid[]) => console.log(results));
I have the following searchService.search method that returns a forkJoin of two api calls.
I want the calls to execute simultaneously which they are but I also want each response back as a single object that can be passed into my SearchSuccess action and processed immediately without waiting for all calls to complete. Currently they are returning as an array of responses and only upon completion of both API calls - as this is what forkJoin is used for.
My issue is that I'm struggling to find another operator that does what I want.
Or perhaps the code pattern requires some redesign?
action:
#Effect()
trySearch: Observable<Action> = this.actions$.pipe(
ofType(SearchActionTypes.TrySearch),
switchMap((action: TrySearch) =>
this.searchService.search(action.payload)
.pipe(
map((data) => new SearchSuccess(data)),
catchError(error => of(new SearchFail(error))),
),
),
);
SearchService (snippet):
search(searchForm: SearchForm): Observable<any> {
const returnArray = [];
if (searchForm.searchClients) {
const searchClientParams = new Search();
searchClientParams.searchPhrase = searchForm.searchPhrase;
searchClientParams.type = SearchType.Client;
const searchClients = this.objectSearch(searchClientParams);
returnArray.push(searchClients);
}
if (searchForm.searchContacts) {
const searchContactParams = new Search();
searchContactParams.searchPhrase = searchForm.searchPhrase;
searchContactParams.type = SearchType.Contact;
const searchContacts = this.objectSearch(searchContactParams);
returnArray.push(searchContacts);
}
return Observable.forkJoin(returnArray);
}
If I understand it correctly returnArray contains two Observables and you want to wait until they both complete but still you want to emit each result separately.
Since forkJoin emits all results in a array you could just unwrap it with mergeMap (or concatMap):
this.searchService.search(action.payload)
.pipe(
mergeMap(results => results),
map((data) => new SearchSuccess(data)),
catchError(error => of(new SearchFail(error))),
),