I continue to struggle with serial/parallel processing in JS (promises). I want to query my server and identify those queries that take longer than, say 500 ms. The following works, but as far as I understand, the queries are made one after another.
const query = async (queries) => {
for (let i = 0, j = queries.length; i < j; i++) {
let t = process.hrtime();
const response = await fetch(queries[i]);
const result = await response.json();
t = process.hrtime(t);
const ms = Math.round((t[0] * 1000) + (t[1] / 1000000));
if (ms > 500) {
console.log(ms, queries[i]);
}
}
}
query(arrayOfQueries);
// console output below (snipped for brevity)
3085 http://localhost:3010/v3/…
2463 http://localhost:3010/v3/…
2484 http://localhost:3010/v3/…
…
I change the above code to fire the queries in parallel, but now I get back an array of promises. I can't figure out how to identify only those promises that take longer than 500ms to resolve
const query = async (queries) => {
const r = await Promise.all(queries.map(async (q) => fetch(q)));
console.log(r);
};
// console output
[
Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },
[Symbol(Response internals)]: {
url: 'http://localhost:3010/v3/…',
status: 200,
statusText: 'OK',
headers: [Headers],
counter: 0
}
},
Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },
[Symbol(Response internals)]: {
url: 'http://localhost:3010/v3/…',
status: 200,
statusText: 'OK',
headers: [Headers],
counter: 0
}
},
Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },
[Symbol(Response internals)]: {
url: 'http://localhost:3010/v3/…',
status: 200,
statusText: 'OK',
headers: [Headers],
counter: 0
}
},
When running the queries in parallel, you would have to add code (similar to what you had for your non-parallel example) to time each one separately so you could track each individual request separately.
The time of each request overlaps so you can't keep track of the timing of each individual request from the outside. Here's an example of timing each individual request:
const query = async (queries) => {
const r = await Promise.all(queries.map(async (q) => {
const start = Date.now();
const response = await fetch(q);
const json = await response.json();
const delta = Date.now() - start;
console.log(`${delta}ms for ${q}`);
return json;
});
return r;
};
This will output the timing for each request at the time it finishes which may not be in the same order that the requests were made. If you want, you can collect these timing results into an array and output all the timing at once at the end.
Related
I'm currently trying to implement pagination on my posts.
Using Apollo graphql here is my useQuery
const { data: postsData, fetchMore } = useQuery(POSTS_BY_USER_DRAFT, {
fetchPolicy: 'network-only',
variables: {
user: user.id,
start: 0,
limit: limit
},
onCompleted: () => {
setTotal(postsData[model].meta.pagination.total)
}})
and here is my onClick handler for fetching more posts
const loadMorePosts = async () => {
const nextStart = start + limit
setStart(nextStart);
await fetchMore({
variables: {
user: user.id,
offset: nextStart,
limit: limit,
},
updateQuery: (prevResult, { fetchMoreResult }) => {
if (!fetchMoreResult) {
return prevResult
}
const prevData = prevResult[model].data
const moreData = fetchMoreResult[model].data
fetchMoreResult[model].data = [...prevData, ...moreData]
// fetchMoreResult[model].data = [...moreData]
return fetchMoreResult
},
})}
My queries are successful as I do get correctly the data, however postsData does not get updated
[NOTICED]: If I switch fetchMoreResult[model].data = [...prevData, ...moreData] for
fetchMoreResult[model].data = [...moreData] my postsData does get updated.
I have tried return { ...fetchMoreResult } and multiple ways of returning data fearing an immutability/comparaison issue but it does not seem to do the job.
I'm not sure why, but setting a fetchPolicy for Apollo will do the job
const client = new ApolloClient({
link: authLink.concat(httpLink),
cache: new InMemoryCache({
typePolicies: {
Publication: {
merge: true,
},
Post: {
merge: true,
},
},
}),
defaultOptions: defaultOptions,
})
I have records in strapi. I am using strapi content API. In my front-end, I need to display only 2 records randomly. For limiting, I have used limit query from content API. But random fetching what keyword I need to use. The official documentation doesn't provide any details regarding this - https://strapi.io/documentation/v3.x/content-api/parameters.html#available-operators
There's no official Strapi API parameter for random. You have to implement your own. Below is what I've done previously, using Strapi v3:
1 - Make a service function
File: api/mymodel/services/mymodel.js
This will contain our actual random query (SQL), and wrapping it in a service is handy because it can be used in many places (cron jobs, inside other models, etc).
module.exports = {
serviceGetRandom() {
return new Promise( (resolve, reject) => {
// There's a few ways to query data.
// This example uses Knex.
const knex = strapi.connections.default
let query = knex('mydatatable')
// Add more .select()'s if you want other fields
query.select('id')
// These rules enable us to get one random post
query.orderByRaw('RAND()')
query.limit(1)
// Initiate the query and do stuff
query
.then(record => {
console.log("getRandom() record: %O", record[0])
resolve(record[0])
})
.catch(error => {
reject(error)
})
})
}
}
2 - Use the service somewhere, like a controller:
File: api/mymodel/controllers/mymodel.js
module.exports = {
//(untested)
getRandom: async (ctx) => {
await strapi.services.mymodel.serviceGetRandom()
.then(output => {
console.log("getRandom output is %O", output.id)
ctx.send({
randomPost: output
}, 200)
})
.catch( () => {
ctx.send({
message: 'Oops! Some error message'
}, 204) // Place a proper error code here
})
}
}
3 - Create a route that points to this controller
File: api/mymodel/config/routes.json
...
{
"method": "GET",
"path": "/mymodelrandom",
"handler": "mymodel.getRandom",
"config": {
"policies": []
}
},
...
4 - In your front-end, access the route
(However you access your API)
e.g. ajax call to /api/mymodelrandom
There is no API parameter for getting a random result.
So: FrontEnd is the recommended solution for your question.
You need to create a random request range and then get some random item from this range.
function getRandomInt(max) {
return Math.floor(Math.random() * Math.floor(max));
}
const firstID = getRandomInt(restaurants.length);
const secondID = getRandomInt(3);
const query = qs.stringify({
id_in:[firstID,secondID ]
});
// request query should be something like GET /restaurants?id_in=3&id_in=6
One way you can do this reliably is by two steps:
Get the total number of records
Fetch the number of records using _start and _limit parameters
// Untested code but you get the idea
// Returns a random number between min (inclusive) and max (exclusive)
function getRandomArbitrary(min, max) {
return Math.random() * (max - min) + min;
}
const { data: totalNumberPosts } = await axios.get('/posts/count');
// Fetch 20 posts
const _limit = 20;
// We need to be sure that we are not fetching less than 20 posts
// e.g. we only have 40 posts. We generate a random number that is 30.
// then we would start on 30 and would only fetch 10 posts (because we only have 40)
const _start = getRandomArbitrary(0, totalNumberPosts - _limit);
const { data: randomPosts } = await axios.get('/posts', { params: { _limit, _start } })
The problem with this approach is that it requires two network requests but for my needs, this is not a problem.
This seem to work for me with Strapi v.4 REST API
Controller, Get 6 random entries
"use strict";
/**
* artwork controller
*/
const { createCoreController } = require("#strapi/strapi").factories;
module.exports = createCoreController("api::artwork.artwork", ({ strapi }) => {
const numberOfEntries = 6;
return {
async random(ctx) {
const entries = await strapi.entityService.findMany(
"api::artwork.artwork",
{
populate: ["image", "pageHeading", "seo", "socialMedia", "artist"],
}
);
const randomEntries = [...entries].sort(() => 0.5 - Math.random());
ctx.body = randomEntries.slice(0, numberOfEntries);
},
};
});
Route
random.js
"use strict";
module.exports = {
routes: [
{
method: "GET",
path: "/artwork/random",
handler: "artwork.random",
config: {
auth: false,
},
},
],
};
API
http://localhost:1337/api/artwork/random
To match default data structure of Strapi
"use strict";
/**
* artwork controller
*/
const { createCoreController } = require("#strapi/strapi").factories;
module.exports = createCoreController("api::artwork.artwork", ({ strapi }) => {
const numberOfEntries = 6;
return {
async random(ctx) {
const entries = await strapi.entityService.findMany(
"api::artwork.artwork",
{
populate: ["image", "pageHeading", "seo", "socialMedia", "artist"],
}
);
const randomEntries = [...entries]
.sort(() => 0.5 - Math.random())
.slice(0, numberOfEntries);
const structureRandomEntries = {
data: randomEntries.map((entry) => {
return {
id: entry.id,
attributes: entry,
};
}),
};
ctx.body = structureRandomEntries;
},
};
});
There is also a random sort plugin.
https://www.npmjs.com/package/strapi-plugin-random-sort
This seem to work for me with Strapi v4.3.8 and graphql
src/index.js
"use strict";
module.exports = {
register({ strapi }) {
const extensionService = strapi.service("plugin::graphql.extension");
const extension = ({ strapi }) => ({
typeDefs: `
type Query {
randomTestimonial: Testimonial
}
`,
resolvers: {
Query: {
randomTestimonial: async (parent, args) => {
const entries = await strapi.entityService.findMany(
"api::testimonial.testimonial"
);
const sanitizedRandomEntry =
entries[Math.floor(Math.random() * entries.length)];
return sanitizedRandomEntry;
},
},
},
resolversConfig: {
"Query.randomTestimonial": {
auth: false,
},
},
});
extensionService.use(extension);
},
bootstrap({ strapi }) {},
};
graphql query:
query GetRandomTestimonial {
randomTestimonial {
__typename
name
position
location
description
}
}
generate random testimonial on route change/refresh
https://jungspooner.com/biography
Im using NestJS. I want to get all data from paginated API (i dont know the total page). Right now im using while loop to get all the data until the API returns 204 No Content, this is my code so far:
async getProduct() {
let productFinal: ProductItem[] = [];
let products: ProductItem[] = [];
let offset = 1;
let state = COLLECTING_STATE.InProgress;
let retryCount = 1;
do {
const path = `product?limit=50&offset=${offset}`;
products = await this.httpService
.get(path, { headers, validateStatus: null })
.pipe(
concatMap((response) => {
// if the statusCode is "204", the loop is complete
if (response.status === 204) {
state = COLLECTING_STATE.Finish;
}
// check if the response is error
if (response.status < 200 || response.status >= 300) {
// log error
Logger.error(
`[ERROR] Error collecting product on offset: ${offset}. StatusCode: ${
response.status
}. Error: ${JSON.stringify(response.data)}. Retrying... (${retryCount})`,
undefined,
'Collect Product'
);
// increment the retryCount
retryCount++;
// return throwError to trigger retry event
return throwError(`[ERROR] Received status ${response.status} from HTTP call`);
}
// return the data if OK
return of(response.data.item);
}),
catchError((err) => {
if (err?.code) {
// log error
Logger.error(
`Connection error: ${err?.code}. Retrying... (${retryCount})`,
undefined,
'Collect Product'
);
// increment the retryCount
retryCount++;
}
return throwError(err);
}),
// retry three times
retry(3),
// if still error, then stop the loop
catchError((err) => {
Logger.error(
`[ERROR] End retrying. Error: ${err?.code ?? err}`,
undefined,
'Collect Product'
);
state = COLLECTING_STATE.Finish;
return of(err);
})
)
.toPromise();
// set retryCount to 1 again
retryCount = 1;
// check if products is defined
if (products?.length > 0) {
// if so, push the product to final variable
productFinal = union(products, productFinal);
}
// increment the offset
offset++;
// and loop while the state is not finish
} while ((state as COLLECTING_STATE) !== COLLECTING_STATE.Finish);
return productFinal;
}
The endpoint product?limit=50&offset=${offset} is from third-party service, it doesn't have one endpoint to grab all the data so this is the only way, it has a maximum limit of 50 per offset, and it didn't have a nextPage or totalPage information on the response so i have to make offset variable and increment it after the previous request is complete.
How do I replace the while loop with the RxJS operator? And can it be optimized to make more than one request at a time (maybe four or five), thus taking less time to get all data?
Based on answer from RxJS Observable Pagination, but increment offset every time request is made:
const { of, timer, defer, EMPTY, from, concat } = rxjs; // = require("rxjs")
const { map, tap, mapTo, mergeMap, take } = rxjs.operators; // = require("rxjs/operators")
// simulate network request
function fetchPage({ limit, offset }) {
// 204 resposne
if (offset > 20) {
return of({ status: 204, items: null });
}
// regular data response
return timer(100).pipe(
tap(() =>
console.log(`-> fetched elements from ${offset} to ${offset+limit}`)
),
mapTo({
status: 200,
items: Array.from({ length: limit }).map((_, i) => offset + i)
})
);
}
const limit = 10;
function getItems(offset = 0) {
return defer(() => fetchPage({ limit, offset })).pipe(
mergeMap(({ status, items }) => {
if (status === 204) {
return EMPTY;
}
const items$ = from(items);
const next$ = getItems(offset + limit);
return concat(items$, next$);
})
);
}
// process only first 100 items, without fetching all of the data
getItems()
.pipe(take(100))
.subscribe({
next: console.log,
error: console.error,
complete: () => console.log("complete")
});
<script src="https://unpkg.com/rxjs#6.6.2/bundles/rxjs.umd.min.js"></script>
Regarding possible optimization to make parallel requests - I don't think it will work well. Instead you could show data progressively, as soon as items are loading. Or change API as was suggested in the comments.
the timeout that I defined does not throw any error when the duration parameter I defined is greater than 7000 ms. what is strange is that the timeout operator works well in my code from 0 to 7000 ms
pay(billing: Billing): Observable {
const httpOptions = {
headers: new HttpHeaders({
// 'Access-Control-Allow-Origin':'*'
}),
params: new HttpParams()
.append('timezone', billing.timezone)
.append('mode', billing.mode)
.append('responseFailURL', billing.responseFailURL)
.append('responseSuccessURL', billing.responseSuccessURL)
.append('hash', billing.hash)
.append('txndatetime', billing.txndatetime)
.append('chargetotal', billing.chargetotal.toString())
.append('storename', billing.storename.toString())
.append('currency', billing.currency.toString())
};
// Sending required payment infrmations to Authipay host url
return forkJoin(
of(2), timer(2000).pipe(mergeMap(value => this.getPayementStatus(billing.txndatetime))).pipe( timeout(7500))
).pipe(
map(
([articles, authorOfTheMonth]) => {
console.log(authorOfTheMonth);
return authorOfTheMonth;
}
)
).subscribe(
resp => {
this.router.navigate(['success'], { relativeTo: this.route });
} else {
form.setErrors({ paymentFailed: true });
this.alertify.error(this.translate.instant('error.payment'));
}
},
error => {
if (error instanceof TimeoutError) {
this.alertify.error(error.message);
} else {
this.alertify.error(this.translate.instant('error.payment'));
}
}
);
timeout seems to work as expected to me.
I wrote a test here where I replaced your this.getPayementStatus(billing.txndatetime)) function with a :
simulated response
const simulateResponseTime = (timeInMS)=> timer(timeInMS); // in milliseconds
Which will return a response in delayOfResponse milliseconds. With this tool we can test what happens when the response takes more time than timeout threshold:
Simulation parameters
const timeoutThreshold = 7500; // in ms
const delayOfResponse = 200; //in ms
Finally, a minimalist version of
Your code
forkJoin(of(2), timer(2000).pipe(
mergeMap(value => simulateResponseTime(delayOfResponse))
).pipe(timeout(timeoutThreshold))
).pipe(
...
).subscribe(
resp => {
console.log('Success')
},
error => {
console.log('Error message :', error.message)
console.log('Error type :', error.name)
console.log('Is a TimeoutError :', error.name === 'TimeoutError' )
}
);
I want to test an effect that works as follows:
Effect starts if LoadEntriesSucces action was dispatched
It waits for 5 seconds
After 5 seconds passes http request is send
When response arrives, new action is dispatched (depending, whether response was succes or error).
Effect's code looks like this:
#Effect()
continuePollingEntries$ = this.actions$.pipe(
ofType(SubnetBrowserApiActions.SubnetBrowserApiActionTypes.LoadEntriesSucces),
delay(5000),
switchMap(() => {
return this.subnetBrowserService.getSubnetEntries().pipe(
map((entries) => {
return new SubnetBrowserApiActions.LoadEntriesSucces({ entries });
}),
catchError((error) => {
return of(new SubnetBrowserApiActions.LoadEntriesFailure({ error }));
}),
);
}),
);
What I want to test is whether an effect is dispatched after 5 seconds:
it('should dispatch action after 5 seconds', () => {
const entries: SubnetEntry[] = [{
type: 'type',
userText: 'userText',
ipAddress: '0.0.0.0'
}];
const action = new SubnetBrowserApiActions.LoadEntriesSucces({entries});
const completion = new SubnetBrowserApiActions.LoadEntriesSucces({entries});
actions$ = hot('-a', { a: action });
const response = cold('-a', {a: entries});
const expected = cold('- 5s b ', { b: completion });
subnetBrowserService.getSubnetEntries = () => (response);
expect(effects.continuePollingEntries$).toBeObservable(expected);
});
However this test does not work for me. Output from test looks like this:
Expected $.length = 0 to equal 3.
Expected $[0] = undefined to equal Object({ frame: 20, notification: Notification({ kind: 'N', value: undefined, error: undefined, hasValue: true }) }).
Expected $[1] = undefined to equal Object({ frame: 30, notification: Notification({ kind: 'N', value: undefined, error: undefined, hasValue: true }) }).
Expected $[2] = undefined to equal Object({ frame: 50, notification: Notification({ kind: 'N', value: LoadEntriesSucces({ payload: Object({ entries: [ Object({ type: 'type', userText: 'userText', ipAddress: '0.0.0.0' }) ] }), type: '[Subnet Browser API] Load Entries Succes' }), error: undefined, hasValue: true }) }).
What should I do to make this test work?
Like mentioned in another answer, one way to test that effect would be by using the TestScheduler but it can be done in a simpler way.
We can test our asynchronous RxJS code synchronously and deterministically by virtualizing time using the TestScheduler. ASCII marble diagrams provide a visual way for us to represent the behavior of an Observable. We can use them to assert that a particular Observable behaves as expected, as well as to create hot and cold Observables we can use as mocks.
For example, let's unit test the following effect:
effectWithDelay$ = createEffect(() => {
return this.actions$.pipe(
ofType(fromFooActions.doSomething),
delay(5000),
switchMap(({ payload }) => {
const { someData } = payload;
return this.fooService.someMethod(someData).pipe(
map(() => {
return fromFooActions.doSomethingSuccess();
}),
catchError(() => {
return of(fromFooActions.doSomethinfError());
}),
);
}),
);
});
The effect just waits 5 seconds after an initial action, and calls a service which would then dispatch a success or error action. The code to unit test that effect would be the following:
import { TestBed } from "#angular/core/testing";
import { provideMockActions } from "#ngrx/effects/testing";
import { Observable } from "rxjs";
import { TestScheduler } from "rxjs/testing";
import { FooEffects } from "./foo.effects";
import { FooService } from "../services/foo.service";
import * as fromFooActions from "../actions/foo.actions";
// ...
describe("FooEffects", () => {
let actions$: Observable<unknown>;
let testScheduler: TestScheduler; // <-- instance of the test scheduler
let effects: FooEffects;
let fooServiceMock: jasmine.SpyObj<FooService>;
beforeEach(() => {
// Initialize the TestScheduler instance passing a function to
// compare if two objects are equal
testScheduler = new TestScheduler((actual, expected) => {
expect(actual).toEqual(expected);
});
TestBed.configureTestingModule({
imports: [],
providers: [
FooEffects,
provideMockActions(() => actions$),
// Mock the service so that we can test if it was called
// and if the right data was sent
{
provide: FooService,
useValue: jasmine.createSpyObj("FooService", {
someMethod: jasmine.createSpy(),
}),
},
],
});
effects = TestBed.inject(FooEffects);
fooServiceMock = TestBed.inject(FooService);
});
describe("effectWithDelay$", () => {
it("should dispatch doSomethingSuccess after 5 seconds if success", () => {
const someDataMock = { someData: Math.random() * 100 };
const initialAction = fromFooActions.doSomething(someDataMock);
const expectedAction = fromFooActions.doSomethingSuccess();
testScheduler.run((helpers) => {
// When the code inside this callback is being executed, any operator
// that uses timers/AsyncScheduler (like delay, debounceTime, etc) will
// **automatically** use the TestScheduler instead, so that we have
// "virtual time". You do not need to pass the TestScheduler to them,
// like in the past.
// https://rxjs-dev.firebaseapp.com/guide/testing/marble-testing
const { hot, cold, expectObservable } = helpers;
// Actions // -a-
// Service // -b|
// Results // 5s --c
// Actions
actions$ = hot("-a-", { a: initialAction });
// Service
fooServiceMock.someMethod.and.returnValue(cold("-b|", { b: null }));
// Results
expectObservable(effects.effectWithDelay$).toBe("5s --c", {
c: expectedAction,
});
});
// This needs to be outside of the run() callback
// since it's executed synchronously :O
expect(fooServiceMock.someMethod).toHaveBeenCalled();
expect(fooServiceMock.someMethod).toHaveBeenCalledTimes(1);
expect(fooServiceMock.someMethod).toHaveBeenCalledWith(someDataMock.someData);
});
});
});
Please notice that in the code I'm using expectObservable to test the effect using the "virtual time" from the TestScheduler instance.
you could use the done callback from jasmine
it('should dispatch action after 5 seconds', (done) => {
const resMock = 'resMock';
const entries: SubnetEntry[] = [{
type: 'type',
userText: 'userText',
ipAddress: '0.0.0.0'
}];
const action = new SubnetBrowserApiActions.LoadEntriesSucces({entries});
const completion = new SubnetBrowserApiActions.LoadEntriesSucces({entries});
actions$ = hot('-a', { a: action });
const response = cold('-a', {a: entries});
const expected = cold('- 5s b ', { b: completion });
subnetBrowserService.getSubnetEntries = () => (response);
effects.continuePollingEntries$.subscribe((res)=>{
expect(res).toEqual(resMock);
done()
})
});
The second notation doesn't work with jasmine-marbles, use dashes instead:
const expected = cold('------b ', { b: completion });
You will need to do 3 things
1- Inside your beforeEach, you need to override the internal scheduler of RxJs as follows:
import { async } from 'rxjs/internal/scheduler/async';
import { cold, hot, getTestScheduler } from 'jasmine-marbles';
beforeEach(() => {.....
const testScheduler = getTestScheduler();
async.schedule = (work, delay, state) => testScheduler.schedule(work, delay, state);
})
2- Replace delay, with delayWhen as follows:
delayWhen(_x => (true ? interval(50) : of(undefined)))
3- Use frames, I am not really sure how to use seconds for this, so I used frames. Each frame is 10ms. So for example my delay above is 50ms and my frame is -b, so that is the expected 10 ms + I needed another 50ms so this equals extra 5 frames which was ------b so as follows:
const expected = cold('------b ', { b: outcome });