I'm working with RxJs and I have to make a polling mechanism to retrieve updates from a server.
I need to make a request every second, parse the updates, emit it and remember its id, because I need it to request the next pack of updates like getUpdate(lastId + 1).
The first part is easy so I just use interval with mergeMap
let lastId = 0
const updates = Rx.Observable.interval(1000)
.map(() => lastId)
.mergeMap((offset) => getUpdates(offset + 1))
I'm collecting identifiers like this:
updates.pluck('update_id').scan(Math.max, 0).subscribe(val => lastId = val)
But this solution isn't pure reactive and I'm looking for the way to omit the usage of "global" variable.
How can I improve the code while still being able to return observable containing just updates to the caller?
UPD.
The server response for getUpdates(id) looks like this:
[
{ update_id: 1, payload: { ... } },
{ update_id: 3, payload: { ... } },
{ update_id: 2, payload: { ... } }
]
It may contain 0 to Infinity updates in any order
Something like this? Note that this is an infinite stream since there is no condition to abort; you didn't give one.
// Just returns the ID as the update_id.
const fakeResponse = id => {
return [{ update_id: id }];
};
// Fakes the actual HTTP call with a network delay.
const getUpdates = id => Rx.Observable.of(null).delay(250).map(() => fakeResponse(id));
// Start with update_id = 0, then recursively call with the last
// returned ID incremented by 1.
// The actual emissions on this stream will be the full server responses.
const updates$ = getUpdates(0)
.expand(response => Rx.Observable.of(null)
.delay(1000)
.switchMap(() => {
const highestId = Math.max(...response.map(update => update.update_id));
return getUpdates(highestId + 1);
})
)
updates$.take(5).subscribe(console.log);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.6/Rx.js"></script>
To define the termination of the stream, you probably want to hook into the switchMap at the end; use whatever property of response to conditionally return Observable.empty() instead of calling getUpdates again.
Related
I am trying to achieve the following with Rxjs: given an array of job ids, for every id in the array, poll an endpoint that returns the status of the job. The status can be either "RUNNING", or "FINISHED". The code should poll jobs one after the other, and continue the polling until the jobs are in the "RUNNING" status. As soon as a job reaches the "FINISHED" status, it should be passed downstream, and excluded from further polling.
Below is a minimal toy case that demonstrates the problem.
const {
from,
of,
interval,
mergeMap,
filter,
take,
tap,
delay
} = rxjs;
const { range } = _;
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input })
.pipe(delay(500));
};
const number$ = from(range(1, 10));
const poll = (number) => interval(5000).pipe(
mergeMap(() => {
return doRequest(number)
}),
tap(console.log),
filter(( {status} ) => status === 'FINISHED'),
take(1)
);
const printout$ = number$.pipe(
mergeMap((number) => {
return poll(number)
})
);
printout$.subscribe(console.log);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.5.5/rxjs.umd.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.21/lodash.min.js"></script>
It does most of what I described; but it polls all endpoints simultaneously rather than one after another. Here, roughly, is the pattern I would like to achieve:
starting with ids: [1, 2, 3]
polling: await request 1 then await request 2 then await request 3
then wait for n seconds; then repeat
after job 2 is finished, send request 1, then send request 3, then wait, then repeat
after job 3 is finished, send request 1, then wait, repeat
after job 1 is finished, complete the stream
I feel that in order to achieve the sending of the requests in sequence, they should be concatMaped; but in the snippet above that's not possible because of the interval that would prevent each polling stream from terminating.
Could you please advise how to modify my code to achieve what I am describing?
If I understand the problem right, I would proceed like this.
First of all I would create a poll function that returns an Observable which notifies after a round of pollings, and it emits an array of all numbers for which the call to doRequest returns 'RUNNING'. Such a function would look something like this
const poll = (numbers: number[]) => {
return from(numbers).pipe(
concatMap((n) =>
doRequest(n).pipe(
filter((resp) => resp.status === 'RUNNING'),
map((resp) => resp.value)
)
),
toArray()
);
};
Then what you need to do is to recursively iterate a call the poll function until the array emitted by the Observable returned by poll is empty.
Recursion in rxjs is obtained typically with the expand operator, and this is the operator which we are going to use also in this case, like this
poll(numbers)
.pipe(
expand((numbers) =>
numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)))
)
)
.subscribe(console.log);
A complete example can be seen in this stackblitz.
UPDATE
If the objective is to notify the job ids which have finished with a polling logic, the structure of the solution remains the same (a poll function and recursivity via expand) but the details are different.
The poll function makes sure we emit all the responses of a polling round and it looks like this:
const poll = (
numbers: number[]
) => {
console.log(`Polling ${numbers}`);
return from(numbers).pipe(
concatMap((n) => doRequest(n)),
toArray()
);
};
The recursion logic makes sure that all jobs that are still with "RUNNING" status are polled again but then we filter only the jobs which are FINISHED and passed them downstream. In other words the logic looks like this
poll(start)
.pipe(
expand((responses) => {
const numbers = responses.filter(r => r.status === 'RUNNING').map(r => r.value)
return numbers.length === 0
? EMPTY
: timer(2000).pipe(concatMap(() => poll(numbers)));
}),
map(responses => responses.filter(r => r.status === 'FINISHED')),
filter(finished => finished.length > 0)
)
.subscribe({
next: responses => console.log(`Job finished ${responses.map(r => r.value)}`),
complete: () => {console.log('All processed')}
});
A working example can be seen in this stackblitz.
Updated: Original answer was not on the right track.
What we want to achieve is that on each go around of the interval we poll all the outstanding jobs in order. We yield up any completed jobs to the output observable and we also omit those completed jobs from subsequent polls.
We can do that by using a Subject instead of a static observable of the job IDs. We start our poll interval and we use withLatestFrom to include the latest list of job IDs. We can then add a tap into the output observable when we get a finished job and update the Subject to omit that job.
To end the poller interval we can create an observable that fires when the array of outstanding jobs is empty and use takeUntil with that.
const number$ = new Subject();
const noMoreNumber$ = number$.pipe(skipWhile((numbers) => numbers.length > 0));
const printout$ = interval(5000).pipe(
withLatestFrom(number$),
switchMap(([_, numbers]) => {
return numbers.map((number) => defer(() => doRequest(number)));
}),
concatAll(),
//tap(console.log),
filter(({ status }) => status === 'FINISHED'),
withLatestFrom(number$),
tap(([{ value }, numbers]) =>
number$.next(numbers.filter((num) => num != value))
),
map(([item]) => item),
takeUntil(noMoreNumber$)
);
printout$.subscribe({
next: console.log,
error: console.error,
complete: () => console.log('COMPLETE'),
});
number$.next([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
The other tweak I would make is to use switchMap instead of mergeMap inside the poller itself. If you use that in combination with fromFetch for performing your HTTP calls then, if there is some long-running HTTP call which gets stuck, on the next poll the previous call will be cancelled before it makes the next HTTP call because switchMap disposes of the previous observable before subscribing to the new one.
Here's a working example:
https://stackblitz.com/edit/js-gxrrb3?devToolsHeight=33&file=index.js
Generates console output looking like this...
TRY this
import { delay, EMPTY, from, of, range } from 'rxjs';
import { concatMap, filter, mergeMap, tap, toArray } from 'rxjs/operators';
const number$ = from(range(1, 3));
const doRequest = (input) => {
const status = Math.random() < 0.15 ? 'FINISHED' : 'RUNNING';
return of({ status, value: input }).pipe(delay(1000));
};
const poll = (jobs: object[]) => {
return from(jobs).pipe(
filter((job) => job['status'] !== 'FINISHED'),
concatMap((job) => doRequest(job['value'])),
tap((job) => {
console.log('polling with................', job);
}),
toArray(),
tap((result) => {
console.log('curent jobs................', JSON.stringify(result));
}),
mergeMap((result) =>
result.length > 0 ? poll(result) : of('All job completed!')
)
);
};
const initiateJob = number$.pipe(
mergeMap((id) => doRequest(id)),
toArray(),
tap((jobs) => {
console.log('initialJobs: ', JSON.stringify(jobs));
}),
concatMap(poll)
);
initiateJob.subscribe({
next: console.log,
error: console.log,
complete: () => console.log('COMPLETED'),
});
There is an array in public users = new BehaviorSubject<User[]>([]).
I want to delete element from this observable and refresh it.
My solution is:
const idRemove = 2;
this.users.next(this.user.getValue().filter((u) => u.id !== idRemove)
But I seem I use wrong way of using RXJS
Toward Idiomatic RxJS
Using subscribe instead of .value.
interface User {
id: number
}
const users$ = new BehaviorSubject<User[]>([
{id:1},
{id:2},
{id:3}
]);
function removeId(idRemove: number) {
users$.pipe(
take(1),
map(us => us.filter(u => u.id !== idRemove))
).subscribe(
users$.next.bind(users$)
);
}
users$.subscribe(us =>
console.log("Current Users: ", us)
);
removeId(2);
removeId(1);
removeId(3);
Output:
Current Users: [ { id: 1 }, { id: 2 }, { id: 3 } ]
Current Users: [ { id: 1 }, { id: 3 } ]
Current Users: [ { id: 3 } ]
Current Users: []
To handle state within RxJS pipes you can use the Scan operator
Useful for encapsulating and managing state. Applies an accumulator (or "reducer function") to each value from the source after an initial state is established -- either via a seed value (second argument), or from the first value from the source.
const { Subject, merge } = rxjs;
const { scan, map } = rxjs.operators;
// This function is used to apply new users to the state of the scan
const usersFn = users => state => users
// This function is used to remove all matching users with the given id from the state of the scan
const removeFn = removeId => state => state.filter(user => user.id !== removeId)
// This Subject represents your old user BehaviorSubject
const users$$ = new Subject()
// This Subject represents the place where this.users.next(this.user.getValue().filter((u) => u.id !== idRemove) was called
const remove$$ = new Subject()
// This is your new user$ Observable that handles a state within its pipe. Use this Observable in all places where you need your user Array instead of the user BehaviorSubject
const user$ = merge(
// When users$$ emits the usersFn is called with the users argument (1. time)
users$$.pipe(map(usersFn)),
// When remove$$ emits the removeFn is called with the removeId argument (1. time)
remove$$.pipe(map(removeFn))
).pipe(
// Either the usersFn or removeFn is called the second time with the state argument (2. time)
scan((state, fn) => fn(state), [])
)
// Debug subscription
user$.subscribe(console.log)
// Test emits
users$$.next([
{id: 1, name: "first"},
{id: 2, name: "second"}
])
remove$$.next(2)
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.4.0/rxjs.umd.min.js"></script>
Ben Lesh (main Contributor of RxJS) wrote an anser about why not to use getValue in RxJS: The only way you should be getting values "out of" an Observable/Subject is with subscribe!
Using getValue() is discouraged in general for reasons explained here even though I'm sure there are exception where it's fine to use it. So a better way is subscribing to get the latest value and then changing it:
this.users
.pipe(take(1)) // take(1) will make sure we're not creating an infinite loop
.subscribe(users => {
this.users.next(users.filter((u) => u.id !== idRemove);
});
I am trying to go a simple step beyond the nest doc example in implementing #Sse() in a controller but I never used rxjs untill now so Im a bit confused.
The flow is :
client send a POST request with a file payload
server (hopefully) sends back the newly created project with a prop status:UPLOADED
client subscribe to sse route described below passing as param the projectId it just received from server
in the meantime server is doingSomeStuff that could take from 10sec to a min. When doingSomeStuff is done, project status is updated in db from UPLOADED to PARSED
My need is for the #Sse decorated function to execute at x interval of time a "status-check" and return project.status (that may or may not have been updated at the time)
My present code :
#Sse('sse/:projectId')
sse(#Param('projectId') projectId: string): Observable<any> {
const projId$ = from(this.projectService.find(projectId)).pipe(
map((p) => ({
data: {
status: p.status,
},
})),
);
return interval(1000).pipe(switchMap(() => projId$));
}
I don't put code of the service here as it a simple mongooseModel.findById wrapper.
My problem is the status returned remains UPLOADED and is never updated. It doesnt seem the promise is reexecuted at every tick. If I console.log inside my service I can see my log being printed only once with the initial project value while I expect to see a new log at each tick.
This is a two-step process.
We create an observable out of the promise generated by this.service.findById() using the from operator in rxjs. We also use the map operator to set the format of the object we need when someone subscribes to this observable.
We want to return this observable every x seconds. interval(x) creates an observable that emits a value after every x milliseconds. Hence, we use this and then switchMap to the projId$ whenever the interval emits a value. The switchMap operator switches to the inner observable whenever the outer observable emits a value.
Please note: Since your server may take 10 sec, to min for doing the operation, you should set the intervalValue accordingly. In the code snippet below, I've set it to 10,000 milli seconds which is 10 seconds.
const intervalValue = 10000;
#Sse('sse/:projectId')
sse(#Param('projectId') projectId: string): Observable < any > {
return interval(intervalValue).pipe(
switchMap(() => this.projectService.find(projectId)),
map((p) => ({
data: {
status: p.status,
}
})));
}
// OR
#Sse('sse/:projectId')
sse(#Param('projectId') projectId: string): Observable < any > {
const projId$ = defer(() => this.service.findById(projectId)).pipe(
map(() => ({
data: {
_: projectId
}
}))
);
return interval(intervalValue).pipe(switchMap(() => projId$));
}
#softmarshmallow
You can watch model changes and use observable stream to send it.
Something like this
import { Controller, Param, Sse } from '#nestjs/common'
import { filter, map, Observable, Subject } from 'rxjs'
#Controller('project')
export class ProjectStatusController {
private project$ = new Subject()
// watch model event
// this method should be called when project is changed
onProjectChange(project) {
this.project$.next(project)
}
#Sse('sse/:projectId')
sse(#Param('projectId') projectId: string): Observable<any> {
return this.project$.pipe(
filter((project) => project.projectId === projectId),
map((project) => ({
data: {
status: project.status,
},
})),
)
}
}
Here is what I am trying to do - step #3 is what I am struggling with.
Set custom interval
trigger an http call (subscribe to it)
Once interval is finished, I want to repeat the above steps indefinitely (until a condition is met)
Steps #1 & #2 are working
setRenewalInterval() {
...
return this.timerSub = interval(CUSTOM_INTERVAL * 1000)
.pipe(
take(1),
map(x => {
if (this.isLoggedIn && !tokenHelper.isTokenExpired()) {
console.log("<Requesting Token Renew>")
this.getNewToken().subscribe(
x => {
console.log("<Token renewed successfully>");
},
err => {
console.log("<Token failed to renew>");
this.setLogOut();
},
() => console.log('<renewal observable completed>')
);
}
else {
console.log("<Not Logged in or Token expired; logging out>");
this.setLogOut();
}
})
)
.subscribe();
}
#3 repeat the above steps after the subscription is completed. Either call setRenewalInterval() or preferably do it inside the setRenewalInterval().
I have looked into rxjs repeat() and expand() but I can't get them to work with the above code. Any help is greatly appreciated, especially a testable/runnable code e.g. jsfiddle.
You're really close, interval(...) is already creating an emission indefinitely however take(1) is causing the entire observable to complete after the first emission.
If you remove the take(1) your code inside the map will be called at your interval forever.
map is a useful function when you want to take values emit in an observable and convert it to a new value synchronously, similar to how Array.map() works. Because you're working with observables inside this map we can use something like mergeMap() which will handle the subscription and piping through of values automatically.
setRenewalInterval() {
// ...
return this.timerSub = interval(CUSTOM_INTERVAL * 1000)
.pipe(
mergeMap(() => {
if (this.isLoggedIn && !tokenHelper.isTokenExpired()) {
return this.getNewToken();
} else {
return this.setLogOut();
}
})
)
.subscribe(x => {
// x is the result of every call to getNewToken() or setLogOut()
});
}
I am using socket.io to send a series of responses to my front-end. The responses are intended to be sequential, but depending on the connection created by socket.io they're not always guaranteed to come in the correct order (https://github.com/josephg/ShareJS/issues/375).
Assuming each response had a sequence field that held a number (shown as the number in the picture above), the observable should emit these responses in order.
If a response is received out of order and a certain amount of time (n) passes without getting any response, I would like for my observable to emit an error, to signal to my front-end to reset the connection.
A really nice problem. Below a snippet with most important parts commented.
// mock ordered values
const mockMessages = Rx.Observable.fromEvent(document.querySelector('#emit'), 'click')
.map((e, index) => ({
index,
timestamp: e.timeStamp
}))
.delayWhen(() => Rx.Observable.timer(Math.random() * 2000)) // distort order
// there is a lot of mutability in `keepOrder`, but all of it
// is sealed and does not leak to outside environment
const keepOrder = timeoutMs => stream =>
Rx.Observable.defer(() => // need defer to support retries on error
stream.scan((acc, v) => {
acc.buffer.push(v)
acc.buffer.sort((v1, v2) => v1.index - v2.index)
return acc
}, {
lastEmitted: -1,
buffer: []
})
.mergeMap(info => {
const emission = []
while (info.buffer.length && info.lastEmitted + 1 === info.buffer[0].index) {
emission.push(info.buffer.shift())
info.lastEmitted += 1
}
return Rx.Observable.of(emission)
})
.switchMap(emissions => {
if (!emissions.length) { // this condition indicates out of order
return Rx.Observable.timer(timeoutMs)
.mergeMapTo(Rx.Observable
.throw(new Error('ORDER_TIMEOUT')))
} else {
return Rx.Observable.from(emissions)
}
})
)
mockMessages
.do(x => console.log('mocked', x.index))
.let(keepOrder(1000)) // decrease timeoutMs to increase error probablity
.do(x => console.log('ORDERED', x.index))
.retryWhen(es => es
.do(e => console.warn('ERROR', e)))
.subscribe()
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.2/Rx.js"></script>
<button id="emit">EMIT</button>