Similar to channels and broadcast channels, can flows also be instantiated and reused at multiple places?
General usage of creating flows is wrapping the logic to emit the data inside the flow's body and is returned.
Snippet :
fun listenToDataChanges() : Flow<T>
{
return flow {
dataSource.querySomeInfo()?.consumeEach {
data->
if (someCondition) {
emit(data)
}
}
}
}
Everytime listenToDataChanges() is called, a new flow instance is created and multiple subscriptions would be made. Instead is it possible to create and reuse the instance to avoid multiple subscriptions?
Yes, you just need to store it in a variable instead of recreating the flow each time
By the way it seems like you could simplify this way:
val customFlow = dataSource.querySomeInfo()?.filter { someCondition }
Related
I'm creating few microservices using nestjs.
For instance I have x, y & z services all interconnected by grpc but I want service x to send updates to a webapp on a particular entity change so I have considered server-sent-events [open to any other better solution].
Following the nestjs documentation, they have a function running at n interval for sse route, seems to be resource exhaustive. Is there a way to actually sent events when there's a update.
Lets say I have another api call in the same service that is triggered by a button click on another webapp, how do I trigger the event to fire only when the button is clicked and not continuously keep sending events. Also if you know any idiomatic way to achieve this which getting hacky would be appreciated, want it to be last resort.
[BONUS Question]
I also considered MQTT to send events. But I get a feeling that it isn't possible for a single service to have MQTT and gRPC. I'm skeptical of using MQTT because of its latency and how it will affect internal message passing. If I could limit to external clients it would be great (i.e, x service to use gRPC for internal connections and MQTT for webapp just need one route to be exposed by mqtt).
(PS I'm new to microservices so please be comprehensive about your solutions :p)
Thanks in advance for reading till end!
You can. The important thing is that in NestJS SSE is implemented with Observables, so as long as you have an observable you can add to, you can use it to send back SSE events. The easiest way to work with this is with Subjects. I used to have an example of this somewhere, but generally, it would look something like this
#Controller()
export class SseController {
constructor(private readonly sseService: SseService) {}
#SSE()
doTheSse() {
return this.sseService.sendEvents();
}
}
#Injectable()
export class SseService {
private events = new Subject();
addEvent(event) {
this.events.next(event);
}
sendEvents() {
return this.events.asObservable();
}
}
#Injectable()
export class ButtonTriggeredService {
constructor(private readonly sseService: SseService) {}
buttonClickedOrSomething() {
this.sseService.addEvent(buttonClickedEvent);
}
}
Pardon the pseudo-code nature of the above, but in general it does show how you can use Subjects to create observables for SSE events. So long as the #SSE() endpoint returns an observable with the proper shape, you're golden.
There is a better way to handle events with SSE of NestJS:
Please see this repo with code example:
https://github.com/ningacoding/nest-sse-bug/tree/main/src
Where basically you have a service:
import {Injectable} from '#nestjs/common';
import {fromEvent} from "rxjs";
import {EventEmitter} from "events";
#Injectable()
export class EventsService {
private readonly emitter = new EventEmitter();
subscribe(channel: string) {
return fromEvent(this.emitter, channel);
}
emit(channel: string, data?: object) {
this.emitter.emit(channel, {data});
}
}
Obviously, channel can be any string, as recommendation use path style.
For example: "events/for/<user_id>" and users subscribed to that channel will receive only the events for that channel and only when are fired ;) - Fully compatible with #UseGuards, etc. :)
Additional note: Don't inject any service inside EventsService, because of a known bug.
#Sse('sse-endpoint')
sse(): Observable<any> {
//data have to strem
const arr = ['d1','d2', 'd3'];
return new Observable((subscriber) => {
while(arr.len){
subscriber.next(arr.pop()); // data have to return in every chunk
}
if(arr.len == 0) subscriber.complete(); // complete the subscription
});
}
Yes, this is possible, instead of using interval, we can use event emitter.
Whenever the event is emitted, we can send back the response to the client.
I have this sync modeled as a Single, and only 1 sync can be running at a time.
I'm trying to subscribe the "job" on a Schedulers.single() which mostly works, but inside the chain there are schedulers hops (to db writes scheduler), which unblocks the natural queue created by single()
Then I looked at flatMap(maxConcurrency=1) but this won't work, as that requires always the same instance. I.e. from what I understand, some sort of a Subject of sync requests, which however is uncomposable as my usecase mostly looks like this
fun someAction1AndSync(): Single<Unit> {
return someAction1()
.flatMap { sync() }
}
fun someAction2AndSync(): Single<Unit> {
return someAction2()
.flatMap { sync() }
}
...
as you can see, its separate sync Single instances :/
Also note someActionXAndSync should not emit until the sync is also done
Basically I'm looking for coroutines Semaphore
I can think of three ways:
use a single thread for whole sync operation (decoupling through queue)
use semaphore to protect sync method from entering multiple times (not recommended, because will block callee)
fast return, when sync is in progress (AtomicBoolean)
There might by other solutions, which I am not aware of.
fast return, when sync is in progress
Also note someActionXAndSync should not emit until the sync is also done
This solution will not queue up sync requests, but will fail fast. The callee must handle the error appropriately
SyncService
class SyncService {
val isSync: AtomicBoolean = AtomicBoolean(false)
fun sync(): Completable {
return if (isSync.compareAndSet(false, true)) {
Completable.fromCallable { "" }.doOnEvent { isSync.set(false) }
} else {
Completable.error(IllegalStateException("whatever"))
}
}
}
Handling
When sync process is already happening, you will receive an onError. This issue must be handled somehow, because the onError will be emitted to the subscriber. Either you are fine with it, or you could just ignore it with onErrorComplete
fun someAction1AndSync(): Completable {
return Single.just("")
.flatMapCompletable {
sync().onErrorComplete()
}
}
use a single thread for whole sync operation
You have to make sure, that the whole sync-process is processed in a single job. When the sync-process is composed of multiple reactive steps on other threads, it could happen, that another sync process is started, while one sync process is already in progress.
How?
You have to have a scheduler with one thread. Each sync invocation must be invoked from given scheduler. The sync operation must complete sync in one running job.
I would use this:
fun Observable<Unit>.forceOnlyOneSubscriber(): Observable<Unit> {
val subscriberCount = AtomicInteger(0)
return doOnSubscribe { subscriberCount.incrementAndGet() }
.doFinally { subscriberCount.decrementAndGet() }
.doOnSubscribe { check(subscriberCount.get() <= 1) }
}
You can always generify Unit using generics if you need.
Is it possible to allow emitting values from a Flux conditionally based on a global boolean variable?
I'm working with Flux delayUntil(...) but not able to fully grasp the functionality or my assumptions are wrong.
I have a global AtomicBoolean that represents the availability of a downstream connection and only want the upstream Flux to emit if the downstream is ready to process.
To represent the scenario, created a (not working) test sample
//Randomly generates a boolean value every 5 seconds
private Flux<Boolean> signalGenerator() {
return Flux.range(1, Integer.MAX_VALUE)
.delayElements(Duration.ofMillis(5000))
.map(integer -> new Random().nextBoolean());
}
and
Flux.range(1, Integer.MAX_VALUE)
.delayElements(Duration.ofMillis(1000))
.delayUntil(evt -> signalGenerator()) // ?? Only proceed when signalGenerator returns true
.subscribe(System.out::println);
I have another scenario where a downstream process can accept only x messages a second. In the current non-reactive implementation we have a Semaphore of x permits and the thread is blocked if no more permits are available, with Semaphore permits resetting every second.
In both scenarios I want upstream Flux to emit only when there is a demand from the downstream process, and I do not want to Buffer.
You might consider using Mono.fromRunnable() as an input to delayUntil() like below;
Helper class;
public class FluxCondition {
CountDownLatch latch = new CountDownLatch(10); // it depends, might be managed somehow
Runnable r = () -> { latch.await(); }
public void lock() { Mono.fromRunnable(r) };
public void release() { latch.countDown(); }
}
Usage;
FluxCondition delayCondition = new FluxCondition();
Flux.range(1, 10).delayUntil(o -> delayCondition.lock()).subscribe();
.....
delayCondition.release(); // shall call this for each element
I guess there might be a better solution by using sink.emitNext but this might also require a condition variable for controlling Flux flow.
According my understanding, in reactive programming, your data should be considered in every operator step. So it might be better for you to design your consumer as a reactive processor. In my case I had no chance and followed the way as I described above
What's the standard means of processing input to an AWS Lambda handler function, when the format of the incoming JSON varies depending on the type of trigger?
e.g. I have a Lambda function that gets called when an object is created in an S3 bucket, or when an hourly scheduled event fires. Obviously, the JSON passed to the handler is formatted differently.
Is it acceptable to overload Lambda handler functions, with the input type defined as S3Event for one signature and ScheduledEvent for the other? If not, are developers simply calling JsonConvert.DeserializeObject in try blocks? Or is the standard practice to establish multiple Lambda functions, one for each input type (yuck!)?
You should use one function per event.
Having multiple triggers for one Lambda will just make things way harder, as you'll end up with a bunch of if/else, switch statements or even Factory methods if you want to apply design patterns.
Now think of Lambda functions as small and maintainable. Think of pieces of code that should do one thing and should do it well. By the moment you start having multiple triggers, you kind of end up with a "Lambda Monolith", as it will have way too many responsibilities. Not only that, you strongly couple your Lambda functions with your events, meaning that once a new trigger is added, your Lambda code should change. This is just not scalable after two or three triggers.
Another drawback is that you are bound to using one language only if you architect it like that. For some use cases, Java may be the best option. But for others, it may be Node JS, Python, Go...
Essentially, your functions should be small enough to be easily maintainable and even rewritten if necessary. There's absolutely nothing wrong with creating one function per event, although, apparently, you strongly disapprove it. Think of every Lambda as a separate Microservice, which scales out independently, has its own CI/CD pipeline and its own suite of tests.
Another thing to consider is if you want to limit your Lambda concurrent executions depending on your trigger type. This would be unachievable via the "One-Lambda-Does-It-All" model.
Stick with one Lambda per trigger and you'll sleep better at night.
This is actually possible by doing the following:
Have the Lambda signature take a Stream rather than the Amazon event type, so we can get the raw JSON message.
Read the JSON contents of the stream as a string.
Deserialize the string to a custom type in order to identify the event source.
Use the event source information to deserialize the JSON a second time to the appropriate type for the event source.
For example:
public async Task FunctionHandler(Stream stream, ILambdaContext context)
{
using var streamReader = new StreamReader(stream);
var json = await streamReader.ReadToEndAsync();
var serializationOptions = new JsonSerializationOptions { PropertyNameCaseInsensitive = true };
var awsEvent = JsonSerializer.Deserialize<AwsEvent>(json, serializationOptions);
var eventSource = awsEvent?.Records.Select(e => e.EventSource).SingleOrDefault();
await (eventSource switch
{
"aws:s3" => HandleAsync(Deserialize<S3Event>(json, serializationOptions), context),
"aws:sqs" => HandleAsync(Deserialize<SQSEvent>(json, serializationOptions), context),
_ => throw new ArgumentOutOfRangeException(nameof (stream), $"Unsupported event source '{eventSource}'."),
});
}
public async Task HandlyAsync(S3Event #event) => ...
public async Task HandleAsync(SQSEvent #event) => ...
public sealed class AwsEvent
{
public List<Record> Records { get; set; }
public sealed class Record
{
public string EventSource { get; set; }
}
}
I'm just getting started with RXJS to see if it can replace my currently manual data streams. One thing I'm trying to port is a situation whereby the last value in the stream is remembered, so future observers will always get the 'current' value as well as subsequent ones. This seems to be fulfilled by BehaviorSubject.
However, I need to do this for a group of entities. For example, I might have data that represents a message from a user:
{ userId: 1, message: "Hello!" }
And I want a BehaviorSubject-like object that'll store the last message for all users. Is this something I can do with RXJS out-of-the-box, or would I need to do it myself? (If so, any pointers would be appreciated).
EDIT: After some more thought, it perhaps seems logical to having an 'incoming' subject, an observer that updates a Map, and then a function which I can call which initialises an Observable from the map values, and merges with the incoming stream...?
I use RxJS with a redux-like state setup. I have a BehaviorSubject that holds the current state, and every time an event/action is fired that current state gets passed through functions that produce a new state, which the subject is subscribed to.
Here's a simplified version of what I use:
export default class Flux {
constructor(state) {
//all resources are saved here for disposal (for example, when hot loading)
//this is just a flux dispatcher
this.dispatcher = new rx.Subject(),
// BehaviorSuject constructed with initial state
this.state = new Rx.BehaviorSubject(state),
}
addStore(store, initialState, feature = store.feature) {
this.dispatcher
.share()
// this is where the "reduction" happens. store is a reducer that
// takes an existing state and returns the new state
.flatMap(({action, payload}) =>
store(this.state.getValue(), action, payload))
.startWith(initialState || {})
.subscribe(this.state)
);
return this;
}
addActions(actions: rx.Subject) {
// actions are fed to the dispatcher
this.resources.push(actions.share().subscribe(this.dispatcher));
return this;
}
}
I create a global Flux object with manages state. Then, for every "feature" or "page" or whatever I wish I add actions and stores. It makes managing state very easy, and things like time-travel are something that are a given with Rx.