RxJS: How to loop and handle multiple http call - rxjs

Question

Im using NestJS. I want to get all data from paginated API (i dont know the total page). Right now im using while loop to get all the data until the API returns 204 No Content, this is my code so far:
async getProduct() {
let productFinal: ProductItem[] = [];
let products: ProductItem[] = [];
let offset = 1;
let state = COLLECTING_STATE.InProgress;
let retryCount = 1;
do {
const path = `product?limit=50&offset=${offset}`;
products = await this.httpService
.get(path, { headers, validateStatus: null })
.pipe(
concatMap((response) => {
// if the statusCode is "204", the loop is complete
if (response.status === 204) {
state = COLLECTING_STATE.Finish;
}
// check if the response is error
if (response.status < 200 || response.status >= 300) {
// log error
Logger.error(
`[ERROR] Error collecting product on offset: ${offset}. StatusCode: ${
response.status
}. Error: ${JSON.stringify(response.data)}. Retrying... (${retryCount})`,
undefined,
'Collect Product'
);
// increment the retryCount
retryCount++;
// return throwError to trigger retry event
return throwError(`[ERROR] Received status ${response.status} from HTTP call`);
}
// return the data if OK
return of(response.data.item);
}),
catchError((err) => {
if (err?.code) {
// log error
Logger.error(
`Connection error: ${err?.code}. Retrying... (${retryCount})`,
undefined,
'Collect Product'
);
// increment the retryCount
retryCount++;
}
return throwError(err);
}),
// retry three times
retry(3),
// if still error, then stop the loop
catchError((err) => {
Logger.error(
`[ERROR] End retrying. Error: ${err?.code ?? err}`,
undefined,
'Collect Product'
);
state = COLLECTING_STATE.Finish;
return of(err);
})
)
.toPromise();
// set retryCount to 1 again
retryCount = 1;
// check if products is defined
if (products?.length > 0) {
// if so, push the product to final variable
productFinal = union(products, productFinal);
}
// increment the offset
offset++;
// and loop while the state is not finish
} while ((state as COLLECTING_STATE) !== COLLECTING_STATE.Finish);
return productFinal;
}
The endpoint product?limit=50&offset=${offset} is from third-party service, it doesn't have one endpoint to grab all the data so this is the only way, it has a maximum limit of 50 per offset, and it didn't have a nextPage or totalPage information on the response so i have to make offset variable and increment it after the previous request is complete.
How do I replace the while loop with the RxJS operator? And can it be optimized to make more than one request at a time (maybe four or five), thus taking less time to get all data?

Answer 1

Based on answer from RxJS Observable Pagination, but increment offset every time request is made:
const { of, timer, defer, EMPTY, from, concat } = rxjs; // = require("rxjs")
const { map, tap, mapTo, mergeMap, take } = rxjs.operators; // = require("rxjs/operators")
// simulate network request
function fetchPage({ limit, offset }) {
// 204 resposne
if (offset > 20) {
return of({ status: 204, items: null });
}
// regular data response
return timer(100).pipe(
tap(() =>
console.log(`-> fetched elements from ${offset} to ${offset+limit}`)
),
mapTo({
status: 200,
items: Array.from({ length: limit }).map((_, i) => offset + i)
})
);
}
const limit = 10;
function getItems(offset = 0) {
return defer(() => fetchPage({ limit, offset })).pipe(
mergeMap(({ status, items }) => {
if (status === 204) {
return EMPTY;
}
const items$ = from(items);
const next$ = getItems(offset + limit);
return concat(items$, next$);
})
);
}
// process only first 100 items, without fetching all of the data
getItems()
.pipe(take(100))
.subscribe({
next: console.log,
error: console.error,
complete: () => console.log("complete")
});
<script src="https://unpkg.com/rxjs#6.6.2/bundles/rxjs.umd.min.js"></script>
Regarding possible optimization to make parallel requests - I don't think it will work well. Instead you could show data progressively, as soon as items are loading. Or change API as was suggested in the comments.