Observable defaulting when not empty - rxjs

Question

I have some code that is reading from a database, iterating each row of data and performing some logic on it, then creating an observable that then writes to the database, adding it to an array (creating an array of observables), so that when the array of observables is subscribed to via forkJoin all the necessary data is written to the database.
This seems to work perfectly fine until the number of observables in the array gets quite large. The amount of rows can be anywhere from 0-6000, so the size of the array can grow up to this. When it does get to this size the observable no longer writes to the database but instead returns the default value from defaultIfEmpty. I'm stumped as to why it works normally with smaller amounts of observables, but suddenly becomes empty on larger amounts...
It might be a little more clear with a code example
function writeToDB() {
// rows taken from the database, n = 0..6000
data = []
// array of observables
observables = []
for (const row of data) {
if (row.age > 20) {
// websocket between service and database, returns an observable
const observable = websocket.put(row).pipe(
o$.catchError((err) => {
return r$.of(err)
}),
o$.defaultIfEmpty({
success: true,
status: 200
})
);
observables.push(observable);
}
}
return forkJoin([...observables]);
}
Using this example works perfectly fine when subscribed to, except with a large data set where the array observables is about 5000 in length. At that point it starts to return the defaultIfEmpty values { success: true, status: 200 } and I cannot workout why... Any help or advice would greatly appreciated.

Answer 1

It's not clear from what you've shown here. Still, if this works with a smaller number of calls, then there's a good chance that websocket exhibits some strange behavior at those numbers.
Something worth trying might be to limit the concurrency on you websocket calls.
function writeToDB(data) {
// data contains rows taken from the database, n = 0..6000
return from(data).pipe(
filter(row => row.age > 20),
map(row => websocket.put(row).pipe(
catchError(err => of(err)),
// last makes sure that mergeAll behaves like forkJoin
last(undefined, {
success: true,
status: 200
})
)),
// mergeAll lets you choose how many can run concurrently
// for example, at most 50 websocket calls are made at
// once here
mergeAll(50),
toArray()
);
}
I prefer map, mergeAll over mergeMap in this case (as I think you're less likely to miss the concurrent aspect of this), but you can use either.
function writeToDB(data) {
// data contains rows taken from the database, n = 0..6000
return from(data).pipe(
filter(row => row.age > 20),
mergeMap(row => websocket.put(row).pipe(
catchError(err => of(err)),
// last makes sure that mergeMap behaves like forkJoin
last(undefined, {
success: true,
status: 200
})
), 50), // <- sneaky! ;)
toArray()
);
}