RxJS: Rate limit and concurrently limit HTTP requests made at different places - rxjs

Question

I am writing a script using an external API that needs to limit the requests based on:
a maximum number of requests per second
a maximum of current requests
I found and achieved have this behavior working for a single request, but I have a workflow where I need to:
deal with pagination
make two different kind of requests to the same API, the second one being dependent of the first one.
The snippet below illustrates the requests workflow, I tried several things with the mergeMap and expand concurrent parameters and some techniques I found here that has I said work well for one request but I am a little bit confused on how to track all the requests to "sync" the limits across all the requests.
* Get a page of 1000 companies and the total number of companies.
*
* #param afterId Optional id of the last fetched company (to get the next ones).
*/
function getCompanies(afterId?: string): Observable<{ count: number; companies: Company[] }> {
const options = { limit: 1000, afterId }
return this.http.post('https://example.com/searches/companies', options)
}
function getAllCompanies(): Observable<Company[]> {
let alreadyFetchedCount = 0
this.getCompanies().pipe(
expand(({ count, companies }) =>
count <= alreadyFetchedCount ? EMPTY : this.getCompanies(companies[companies.length - 1].uuid)
),
tap(({ companies }) => (alreadyFetchedCount += companies.length))
)
}
/**
* Get a page of 1000 funding rounds.
*
* #param companyUuid The funding rounds company uuid.
* #param afterId Optional id of the last fetched company (to get the next ones).
*/
function getFundingRounds(
companyUuid: string,
afterId?: string
): Observable<{ count: number; fundingRounds: FundingRound[] }> {
const options = { limit: 1000, companyUuid, afterId }
return this.http.post('https://example.com/searches/companies', options)
}
function getAllFundingRounds(companyUuid: string): Observable<FundingRound[]> {
let alreadyFetchedCount = 0
this.getFundingRounds().pipe(
expand(({ count, fundingrounds }) =>
count <= alreadyFetchedCount ? EMPTY : this.getFundingRounds(fundingrounds[fundingrounds.length - 1].uuid)
),
tap(({ fundingrounds }) => (alreadyFetchedCount += fundingrounds.length)),
reduce((acc, value) => [...acc, ...value], [])
)
}
function main() {
getAllCompanies().pipe(
// Here I get a stream of 1000 or less companies until all companies have been fetched.
// Let's work one company by one company.
mergeMap((companies) => companies),
// For each company, get the funding rounds and return the company extended with them.
mergeMap((company) =>
getAllFundingRounds(company.uuid).pipe(map((fundingRounds) => ({ ...company, fundingRounds })))
),
toArray(),
tap(companies =>
// Do something with the result
)
)
}

Answer 1

You can make use of delay that will make sure that you are making http requests within specified time and concatMap that will help you make requests in sequence
this.function1().pipe(
mergeAll(),
delay(1000),
concatMap((data: any) => this.function2(data.id)),
).subscribe(
console.log
);
function2(id: string = ''): Observable<any> {
console.log('id', id);
return serviceCAll().pipe(
delay(1000)
);
}