RxJS v4 used to have an Observable.transduce method which took a transducer. This allowed the use of library-independent transducer operators which had major performance benefits in the past.
Sources
https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/operators/transduce.md
https://github.com/Reactive-Extensions/RxJS/blob/master/doc/gettingstarted/transducers.md
https://medium.com/front-end-hacking/rxjs-transducers-vs-method-chaining-performance-87561cf4ce65
https://github.com/ReactiveX/rxjs/pull/1323
RxJS v5.5 and v6 have pipeable operators and v6 removed method chaining. Because of this, I assumed RxJS operators were standard transducers. Looking through the source code, that doesn't seem to be the case.
RxJS v6 operators function like a transducer where each value is passed entirely through the chain before the next value goes through, but RxJS v6 operators aren't using the standard transducer methods I've seen in other libraries meaning, I don't think they're portable.
The whole thing about transducers is they don't know anything about the collection itself. Instead of writing 100 operators specifically for observables, you could write 100 operators universally able to be applied to any collection or stream type.
Is .pipe unanimous with .transduce or was this method completely removed in RxJS v5?
I had the exact same question and could not find the answer anywhere. Yes you can pipe, but I believe that would create intermediary observables for each operator. I don't know for sure though, it would be about reading the code.
So I came up with my own transduce operator :
function transformForObserver(o) {
return {
"##transducer/init": function() {
return o;
},
"##transducer/step": function(obs, input) {
return obs.next(input);
},
"##transducer/result": function(obs) {
return obs.complete();
}
};
}
const transduce = (obs, transducer) => {
const xform = transducer(transformForObserver);
return Observable.create(o => {
return obs.subscribe({
next: x => {
const res = tryCatch(
xform["##transducer/step"],
err => {
console.error(`Error occurred in transducer/step!`, err);
return err;
}
)(xform, o, x);
if (res instanceof Error) { o.error(res); }
},
error: err => {
console.error(`Error occurred in observable passed to Rx transduce fn!`, err);
o.error(err);
},
complete: () => {o.complete();}
});
});
}
Haven't tested it yet, will post soon about it if there is interest.
Update : I forked jslongser's tranducers library and included such transducers in it. Fork is https://github.com/brucou/transducers.js, and the function is transduceLazyObservable. Cf. tests for example of use.
Related
I'm using RxJS 7, and I would like to have a child generator (another Observable) emitting values based on the parent data.
I was already able to achieve this, but the solution I found is not efficient in terms of CPU usage because it needs to build a new RxJS pipeline for every parent item, and I believe I'm not using here the full potential RxJS has.
Constraints:
Emitted values from the Parent needs to be available to child generator;
Parent needs to know when child flow is done;
Child Observable can have many operators;
Efficient!
The working example:
const { from, mergeMap, reduce, lastValueFrom } = rxjs
function run() {
const parentData = [{ parentId: 1 }, { parentId: 2 }, { parentId: 3 }]
from(parentData)
.pipe(mergeMap((parent) => lastValueFrom(getChildFlow(parent))))
.subscribe((parent) => console.log(parent))
}
function getChildFlow(parent) {
return from(childGenerator(parent))
.pipe(reduce((acc, value) => {
acc.inner.push(value)
return acc
}, { inner: [] }))
}
async function* childGenerator(parentData) {
for await (const index of [1, 2, 3]) {
yield { childId: index, ...parentData }
}
}
run()
<script src="https://unpkg.com/rxjs#^7/dist/bundles/rxjs.umd.min.js"></script>
The reason I'm looking for a more efficient implementation is because it's intended for a data intensive system which can have millions of items flowing.
Questions!
Does RxJS provide some operator to cover this scenario in a more efficient implementation? I really dug RxJS's documentation and didn't found anything, but I may have missed it.
Would it be possible to reuse the flow on the above implementation? The tricky part here is that the child generator needs to have the parent data.
PS: Don't mind the implementation details of the code above, it's just an example of what I'm trying to achieve, and doesn't cover all the precautions and additional steps I have to justify the use-case.
I found the solution to my problem.
It required using mergeMap, groupBy, reduce and zip.
I'm not convinced it's the best solution, so if you find another approach for this that you think is more efficient, I will certainly upvote your answer and mark it as correct answer over mine.
const { from, mergeMap, tap, zip, map, groupBy, reduce } = rxjs
function run() {
const parent$ = from([{ parentId: 1 }, { parentId: 2 }, { parentId: 3 }])
.pipe(tap(doWhatever))
const reducer = reduce(accumulator, [])
const child$ = parent$
.pipe(mergeMap(childGenerator))
.pipe(tap(doWhatever))
.pipe(groupBy((p) => p.parentId))
.pipe(mergeMap((group$) => group$.pipe(reducer)))
zip([parent$, child$])
.pipe(map((results) => ({ ...results[0], inner: results[1] })))
.pipe(tap(doWhatever))
.subscribe(console.log)
}
function accumulator(acc, cur) {
return [...acc, cur]
}
function doWhatever() {}
async function* childGenerator(parentData) {
for await (const index of [1, 2, 3]) {
yield { childId: index, ...parentData }
}
}
run()
<script src="https://unpkg.com/rxjs#^7/dist/bundles/rxjs.umd.min.js"></script>
What is the best way to collect some specific property from all the leafs of the GraphQL graph, reducing it to some single array? For example, my service functions can "throw" some arbitrary string warnings which I want to collect and supply to the client besides the main data, e.g. expected output:
type EntityOutput {
entity: Entity
warnings: [String!]
}
Resolver:
#Mutation()
async updateEntity(
#Args('id', ParseUUIDPipe) id: string,
#Args('data') input: UpdateDto
): Promise<EntityOutputDto>
{
return {
entity: await this.service.update(id, input),
warnings: [] // ???
};
}
Service method:
async update(id: string, input: UpdateDto): Promise<Entity> {
const entity = await this.repository.findOneOrFail(id, { relations: ['type'] }); // check existence
if (Object.values(input).some(v => v !== undefined)) {
const updateData: Partial<Entity & UpdateDto> = Object.assign({ id }, input);
if (input.isCurrentEntityOfItsType === true) {
await this.typesService.update(entity.type.id, { currentEntityId: id }); // <-- this also can create its own warnings
} else if (input.isCurrentEntityOfItsType === false) {
await this.typesService.update(entity.type.id, { currentEntityId: null as any });
}
await this.repository.save(updateData);
} else {
console.warn(`No properties to change were been provided`); // <-- this is a warning I want to save
}
return this.findOne(id);
}
I think my question can be splitted into 2:
To collect warnings from the service, i.e., in general case, the function calls stack of arbitrary depth. It actually looks more like a general programming problem than a NestJS thing
But even when one implement the feature from the first paragraph the NestJS will walk along the GraphQL graph by itself and there can be additional logs in nested fields.
The solution in its complete general form probably will be over-complicated but at least can anyone suggest the good design for the case represented by the example code?
I have a couple of thoughts:
Should every function in the service return its warnings alongside its main response (for example, in a tuple) so we can incrementally "fold" the array of warnings while "unfolding" the calls stack?
Maybe it would be better to implement using some decorator by which we will mark our service methods?
Maybe RxJS – the NestJS beloved one – can offer us some solution? (I don't know a lot about this library/their philosophy)
Actually the default form of the NestJS output is already looking similar to what I want, it's a JSON with 2 root properties: "errors" and "data". And they can be automatically sent to you simultaneously if the error happened is not so fatal to proceed. Can we somehow overwrite the default response object schema and place warnings there?
The whole question is heavily inspired by this SO discussion but it unfortunately says nothing about the actual possible implementation.
So I've implemented a custom context factory which is executed automatically on every GraphQL request and constructs the object of desired format:
app.module.ts:
export interface AppContext {
warnings: string[];
}
const contextFactory: ContextFunction<any, AppContext> = () => ({
warnings: []
});
Now we can benefit from our newly created interface to add strong typings whenever we reference the context, e.g.:
some.resolver.ts
#Mutation()
async remove(
#Args('id', ParseUUIDPipe) id: string,
#Context() ctx: AppContext
): Promise<FindOneDto>
{
return new FindOneDto(await this.service.remove(id, ctx.warnings));
}
Here the service can add its own warnings to the context.
To collect all of them and return to the API caller I override formatResponse function and append the warnings to the extensions (this is a special GraphQL meta-field serving the developing purposes):
app.module.ts:
const graphqlConfig: GqlModuleOptions = {
context: contextFactory,
formatResponse: (
response: GraphQLResponse | null,
context: GraphQLRequestContext<AppContext>,
): GraphQLResponse =>
{
const warnings = context.context.warnings;
if (warnings.length) {
if (response) {
const extensions = response.extensions || (response.extensions = {});
extensions.warnings = warnings;
} else {
return { extensions: { warnings } };
}
}
return response || {};
},
...
}
Similar approach is used in the official Apollo extension example: https://github.com/apollographql/apollo-server/blob/main/packages/apollo-tracing/src/index.ts.
The only drawback I see now is that injecting the context in resolver's arguments breaks the compliance with auto-generated TypeScript interfaces (I use schema-first approach). In such case, we can switch to per-request-based mode so our resolver/service class instance will be created individually for each and every new request: https://docs.nestjs.com/fundamentals/injection-scopes. Now we can access a context right in the methods without introducing any additional parameters. But this comes with increased latencies and, perhaps, memory-consumption. Another approach will be to create a standalone Nest interceptor.
In this code:
impl Msg {
fn apply_to(&self, state: &mut State) {
match (self, state) {
(Msg::MsgA(m), State::StateOne(s)) => {
m.apply_to_state_one(s);
},
(Msg::MsgB(m), State::StateOne(s)) => {
m.apply_to_state_one(s);
},
// FIXME: can these two dispatches be made into one
(Msg::MsgC(m), State::StateOne(s)) => {
m.apply_to_common_state(&mut s.common);
},
(Msg::MsgC(m), State::StateTwo(s)) => {
m.apply_to_common_state(&mut s.common);
},
(Msg::MsgD(m), State::StateTwo(s)) => {
m.apply_to_state_two(s);
},
(_, _) => { // don't care
()
}
}
}
}
there is unappealing boilerplate, more in the full playground.
In full: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=acca055b435ae21d141eaac70e097e72
I want each Msg to be able to be applied to a StateOne, a StateTwo or either (by way of their common field, common).
Msgs should error when applied to a state for which they don't have an implementation.
I would prefer that all the logic was in the individual messages, with none in Msg or the States.
How can I best express this in Rust's type system?
Update: I'm particularly keen to know if my general approach to this problem is correct for Rust, or whether there are better solutions if either the messages or states were generics or dyn trait objects.
You could consider using struct destructuring in order to combine them:
(Msg::MsgC(m), State::StateOne(StateOne {mut common, ..}))
| (Msg::MsgC(m), State::StateTwo(StateTwo {mut common, ..})) => {
m.apply_to_common_state(&mut common);
},
It isn't the prettiest, but after or patterns get stabilized it might be able to be reduced into (Msg::MsgC(m), State::StateOne(StateOne {mut common, ..}) | State::StateTwo(StateTwo {mut common, ..})).
This is a pretty simple question.
How to implement subscriptions in graphql?
I'm asking specifically for when using graphql.js constructors like below ?
I could not find a clean/simple implementation.
There is another question here, but it deals with relay.js - i don't want to unnecessarily increase the nr of external dependencies in my app.
What i have:
module.exports = function (database){
return new GraphQLSchema(
{ query: RootQuery(database)
, mutation: RootMutation(database)
, subscription: RootSubscription(database) -- i can see this in graphiql - see below
}
);
}
function RootSubscription(database){
return new GraphQLObjectType(
{ name: "RootSubscriptionType"
, fields:
{ getCounterEvery2Seconds:
{ type: new GraphQLNonNull(GraphQLInt)
, args :
{ id: { type: GraphQLString }
}
, subscribe(parent, args, context){
// this subscribe function is never called .. why?
const iterator = simpleIterator()
return iterator
}
}
}
}
)
}
I learned that i need a subscribe() which must return an iterator from this github issue.
And here is a simple async iterator. All this iterator does - is to increase and return the counter every 2 seconds. When it reaches 10 it stops.
function simpleIterator(){
return {
[ Symbol.asyncIterator ]: () => {
let i = 0
return {
next: async function(){
i++
await delay(2000)
if(i > 10){
return { done: true }
}
return {
value: i,
done: false
}
}
}
}
}
}
When i run the graphiql subscription, it returns null for some reason:
I'm piecing together code from multiple sources - wasting time and hacking it basically. Can you help me figure this one out?
Subscriptions are such a big feature, where are they properly documented? Where is that snippet of code which you just copy paste - like queries are for example - look here.
Also, i can't use an example where the schema is separate - as a string/from a file. I already created my schema as javascript constructors. Now since im trying to add subscriptions i can't just move back to using a schema as a string. Requires rewriting the entire project. Or can i actually have both? Thanks :)
What problems could arise when using variables that are external to an observable sequence inside the sequence?
For example:
updateCar(newCar: any): Observable<any> {
return of(...).pipe(
switchMap(
(value: any) => {
if (newCar.has4Wheels && value.lovePizza) {
// return a 4 wheel observable
} else {
// return a not 4 wheel observable
}
}
),
switchMap(
(value: any) => {
if (newCar.has4Windows && !value.lovePizza) {
// return a 4 window observable
} else {
// return a 2 window observable
}
}
)
);
}
I know the example above is weird, but i am just using it to ask the question.
What problems could arise with using newCar inside the sequence like being used in the example when it is external to the sequence? If there are no problems, great! Just feels like there is something wrong with this usage to me.
I think nothing (at least as far as you don't modify newCar).
It's true that you could rewrite this and start with for example the following:
of([whatever, newCar])
.pipe(
switchMap(([whatever, newCar]) => {
...
})
)
...
But I think this isn't necessary and would make things just more complicated without any real benefit.