I have the following sample code that recursively pulls the data from some REST resource.
Each endpoint serves the following JSON content
{ data: '123', urls: ['https//example/bar', 'https//example/bim'] }
I can build an Rx stream that pulls the content recursively like this:
function getData(url) {
return fromFetch(url).pipe(
mergeMap(resp => resp.json()),
mergeMap(content => merge(of(content.data), ...content.urls.map(getData)))
)
}
My question is what would be the best way to control the concurrency of HTTP calls here.
Can this be achieved through a custom Scheduler?
or should I simply wrap fromFetch into a class that controls the concurrency of how many HTTP calls can be in flight at any point in time?
With expand you probably only get maximum 1 concurrency, it is easier to control concurrency with subject + mergeMap so that you only have a single channel to work on.
const fetchSubject=new Subject();
fetchSubject.pipe(
mergeMap(url=>fromFetch(url),null,CONCURRENCY)
mergeMap(resp => resp.json()),
tap(url=>content.urls.map(url=>fetchSubject.next(url))
scan((acc,curr) =>{
return [curr.data,...acc]
})
)
Related
So on Cypress's documents it mentions (here: https://docs.cypress.io/guides/core-concepts/variables-and-aliases#Sharing-Context):
Keep in mind that there are use cases for both approaches because they
have different ergonomics.
When using this.users we have access to it synchronously, whereas when
using cy.get('#users') it becomes an asynchronous command.
You can think of the cy.get('#users') as doing the same thing as
cy.wrap(this.users).
This is in regards to using this.fixture vs using cy.get('#fixture') for example (I realize there are other use cases but lets just stick with fixtures for now)
I realize it explains sync vs async above....but when would you WANT to use a synchonous this to access a fixture vs not? Maybe within a then function/callback? (If you needed to access a fixture within a response to a request or something?
One thing it does is reduce callback nesting when multiple aliases are required in one expression, for example
cy.wrap('1').as('a')
// more actions
cy.wrap('2').as('b')
// more actions
cy.wrap('3').as('c')
// assert all values
cy.get('#a').then(a => {
cy.get('#b').then(b => {
cy.get('#c').then(c => {
expect(a+b+c).to.eq(d)
})
})
})
compare to
cy.wrap('1').as('a')
// more actions
cy.wrap('2').as('b')
// more actions
cy.wrap('3').as('c')
// assert all values
cy.then(function() {
expect(this.a + this.b + this.c).to.eq(d)
})
My situation is as follows: I am performing sequential HTTP requests, where one HTTP request depends on the response of the previous. I would like to combine the response data of all these HTTP requests into one observable. I have implemented this before using an async generator. The code for this was relatively simple:
async function* AsyncGeneratorVersion() {
let moreItems = true; // whether there is a next page
let lastAssetId: string | undefined = undefined; // used for pagination
while (moreItems) {
// fetch current batch (this performs the HTTP request)
const batch = await this.getBatch(/* arguments */, lastAssetId);
moreItems = batch.more_items;
lastAssetId = batch.last_assetid;
yield* batch.getSteamItemsWithDescription();
}
}
I am trying to move away from async generators, and towards RxJs Observables. My best (and working) attempt is as follows:
const observerVersion = new Observable<SteamItem>((subscriber) => {
(async () => {
let moreItems = true;
let lastAssetId: string | undefined = undefined;
while (moreItems) {
// fetch current batch (this performs the HTTP request)
const batch = await this.getBatch(/* arguments */, lastAssetId);
moreItems = batch.more_items;
lastAssetId = batch.last_assetid;
const items = batch.getSteamItemsWithDescription();
for (const item of items) subscriber.next(item);
}
subscriber.complete();
})();
});
Now, I believe that there must be some way of improving this Observer variant - this code does not seem very reactive to me. I have tried several things using pipe, however unfortunately these were all unsuccessful.
I found concatMap to come close to a solution. This allowed me to concat the next HTTP request as an observable (done with the this.getBatch method), however I could not find a good way to also not abandon the response of the current HTTP request.
How can this be achieved? In short I believe this problem could be described as appending data to an observable inside the observable itself. (But perhaps this is not a good way of handling this situation)
TLDR;
Here's a working StackBlitz demo.
Explanation
Here would be my approach:
// Faking an actual request
const makeReq = (prevArg, response) =>
new Promise((r) => {
console.log(`Running promise with the prev arg as: ${prevArg}!`);
setTimeout(r, 1000, { prevArg, response });
});
// Preparing the sequential requests.
const args = [1, 2, 3, 4, 5];
from(args)
.pipe(
// Running the reuqests sequantially.
mergeScan(
(acc, crtVal) => {
// `acc?.response` will refer to the previous response
// and we're using it for the next request.
return makeReq(acc?.response, crtVal);
},
// The seed(works the same as `reduce`).
null,
// Making sure that only one request is run at a time.
1
),
// Combining all the responses into one object
// and emitting it after all the requests are done.
reduce((acc, val, idx) => ({ ...acc, [`request${idx + 1}`]: val }), {})
)
.subscribe(console.warn);
Firstly, from(array) will emit each item from the array, synchronously and one by one.
Then, there is mergeScan. It is exactly the result of combining scan and merge. With scan, we can accumulate values(in this case we're using it to have access to the response of the previous request) and what merge does is to allow us to use observables.
To make things a bit easier to understand, think of the Array.prototype.reduce function. It looks something like this:
[].reduce((acc, value) => { return { ...acc }}, /* Seed value */{});
What merge does in mergeScan is to allow us to use the accumulator something like (acc, value) => new Observable(...) instead of return { ...acc }. The latter indicates a synchronous behavior, whereas with the former we can have asynchronous behavior.
Let's go a bit step by step:
when 1 is emitted, makeReq(undefined, 1) will be invoked
after the first makeReq(from above) resolves, makeReq(1, 2) will be invoked
after makeReq(1, 2) resolves, makeReq(2, 3) will be invoked and so on...
Somebody I consulted regarding this matter came up with this solution, I think it's quite elegant:
defer(() => this.getBatch(options)).pipe(
expand(({ more_items, last_assetid }) =>
more_items
? this.getBatch({ ...options, startAssetId: last_assetid })
: EMPTY,
),
concatMap((batch) => batch.getSteamItemsWithDescription()),
);
From my understanding the use of expand here is very similar to the use of mergeScan in #Andrei's answer
I'm using rxjs with NodeJS in backend.
I have a Rest API which allow consumers to run remote yarn installation process. The install function returns an observable of the process. So when the module is installed successfully it emits a value in the observable and complete. At this point, the Rest API will returns a response to the user to say that the installation is successful. In case that the installation fails, the process will throw an Error in the stream and the Rest API returns another response with the error information.
My issue is:
The API is called multiple times in parallel by consumers, so there will be a parallel installations in the backend.
I tried throttle operator to create a queue but it keeps the first stream active. So if the first process is "completed", it returns "true" but the stream doesn't complete
export class MyService {
// the function called by the REST API
installGlobal(moduleName: string): Observable < boolean > {
// I think, there are something to do here to make it queuing
return this.run('yarn', ['global', 'add', moduleName]);
}
private run(cmd: string, args: string[]): Observable < boolean > {
const cmd$ = fromPromise(spawn(cmd, args)).pipe(
map(stdout => {
this.logger.info(`Install Module Successfully`);
this.logger.info(`stdout: ${stdout.toString()}`);
return true;
}),
catchError(error => {
const errorMessage: string = error.stderr.toString();
return _throw(errorMessage.substr(errorMessage.indexOf(' ') + 1));
})
);
return cmd$;
}
}
My expectation:
Either there are multiple request, they must be queued. So the first one will be treated and all parallel onces must be queued. When the first is processed, it must returns the response to the API consumers (like 200 completed) and resume the next stream from the queue.
[UPDATE-01 July 2019]: adding an example
You can fetch a demo of the code at stackblitz
I have reimplemented the existant code and i'm simulating my API call by subscribing multi time to the service which will call the queue
A simple queque in Rxjs can be done like below
const queque=new Subject()
// sequential processing
queue.pipe(concatMap(item=>yourObservableFunction(item)).subscribe()
// add stuff to the queue
queque.next(item)
I'am currently switched from promises to observables. I am using Redux-Observable for my react app. Basically, I am looking for the best operator that will enable mutliple, concurrent ajax calls and return the responses when all the observables have sucessfully finished executing.
Here is a code snippet from my app.
let epicPostAd = (action$, store, {ajax}) =>
action$.ofType(POST_AD)
.debounceTime(1000)
.mergeMap(({ payload }) =>
ajax(generateAjaxRequestSetting(POSTS_URL, 'post', payload,CT_JSON))
.map(response => postAdSuccessful(response))
.catch(e => Observable.of(postAdUnsuccessful(e.xhr.response)))
.takeUntil(action$.ofType(LOCATION_CHANGE))
)
It is a simple ajax request that posts given ad and dispatches POST_AD_SUCCESSFUL when response is 201 else dispatches POST_AD_UNSUCCESSFUL on error.
But the issues is I want to make subsequent ajax observable stream when there is a response. Such as
.map(response => /* start a stream of ajax observables then process the response */)
I will appreciate if you show me the optimal way of achieving this.
Sounds like you're looking for the forkJoin operator.
It will subscribe to all the Observables you pass to it and after they all complete, it will emit the last value from each inside an array.
It wasn't entirely clear where in your Epic you wanted to do this, so I just made a generic example:
const somethingEpic = (action$, store, { ajax }) =>
action$.ofType(SOMETHING)
.mergeMap(() =>
Observable.forkJoin(
ajax('/first'),
ajax('/second'),
ajax('/third')
)
.do(results => {
// the results is an array, containing each
const [first, second, third] = results;
console.log(first, second, third);
})
.map(results => ({
type: 'SOME_RESULTS',
results
}))
);
Technically, it supports a final resultSelector argument you can use instead of using the map operator after it, but I tend not to use it because I've found it's less clear with only negligible performance benefits in common redux-observable style cases. But it's still good to know. Can be handy for more "data-normalization" stuff rather than "transform this into an action" stuff.
const somethingEpic = (action$, store, { ajax }) =>
action$.ofType(SOMETHING)
.mergeMap(() =>
Observable.forkJoin(
ajax('/first'),
ajax('/second'),
ajax('/third'),
results => ({
type: 'SOME_RESULTS',
results
})
)
);
ALSO, if you're asking yourself "what operator do I use?" you should try the operator wizard located in the documentation: http://reactivex.io/rxjs/
Scroll down to the part that says:
Do you need to find an operator for your problem? Start by choosing an option from the list below:
I have one existing Observable, and...
I have some Observables to combine together as one Observable, and...
I have no Observables yet, and...
Hint: open your DevTools to experiment with RxJS.
Though in this case, forkJoin is correctly suggested but when you click on it, it isn't yet documented :sadface: But a google search would present many different websites explaining what it does and how to use it (in RxJS and in other Rx implementations in other languages). Like this helpful website
Here is the answer to my own question. Although JayPhelps answered, I realized that my question was not so clear. Using Jay's recommendation. I came up with the following:
let epicPostAd = (action$, store, {ajax, observable}) =>
action$.ofType(POST_AD)
.debounceTime(1000)
.mergeMap(({ payload }) =>
ajax(generateAjaxRequestSetting(POSTS_URL, 'post', payload, CT_JSON))
.mergeMap(response =>
observable.forkJoin(
ajax(''),
ajax('')
)
.do(res => {
const [first, second, third] = results;
console.log(first, second, third);
})
.map(res => postAdSuccessful(res))
)
.catch(e => observable.of(postAdUnsuccessful(e.xhr.response)))
.takeUntil(action$.ofType(LOCATION_CHANGE))
)
So here how it works. I make a post request and immediately after ajax request finishes execution I .mergeMap the response to a stream of ajax ovservables using .forkJoin(). Then process the results
I am trying to get time expiry cache to work for an observable that abstracts a "request-response", using postMessage and message events on the window.
The remote window expects a message getItemList and replies to it with a message of type {type: 'itemList', data: []}.
I would like to model the itemList$ observable in such a way that it caches the last result for 3 seconds, so that no new requests are made during that time, however, I cannot think of a way to achieve that in an elegant (read, one observable – no subjects) and succint manner.
Here is the example in code:
const remote = someIframe.contentWindow;
const getPayload = message => message.data;
const ofType = type => message => message.type === type;
// all messages coming in from the remote iframe
const messages$ = Observable.fromEvent(window, 'message')
.map(getPayload)
.map(JSON.parse);
// the observable of (cached) items
const itemList$ = Observable.defer(() => {
console.log('sending request');
// sending a request here, should happen once every 3 seconds at most
remote.postMessage('getItemList');
// listening to remote messages with the type `itemList`
return messages$
.filter(ofType('itemList'))
.map(getPayload);
})
.cache(1, 3000);
/**
* Always returns a promise of the list of items
* #returns {Promise<T>}
*/
function getItemList() {
return itemList$
.first()
.toPromise();
}
// poll every second
setInterval(() => {
getItemList()
.then(response => console.log('got response', response));
}, 1000);
I am aware of the (very similar) question, but I am wondering if anyone can come up with a solution without explicit subjects.
Thank you in advance!
I believe you are looking for the rxjs operator throttle:
Documentation on rxjs github repo
Returns an Observable that emits only the first item emitted by the
source Observable during sequential time windows of a specified
duration.
Basically, if you would like to wait until the inputs have quieted for a certain period of time before taking action, you want to debounce.
If you do not want to wait at all, but do not wish to make more than 1 query within a specific amount of time, you will want to throttle. From your use case, I think you want to throttle