I have a springboot API which is dealing with lot of processes in backend. I need to stream the status to the frontend. Since I am new to springboot can anyone help me how to achieve this scenario.
Note - Application is going to be containerized in future and I cannot use any cloud service for this.
As there is not much to go off of I will try my best:
If you are using Log4j2 you could simply usethe SocketAppender (external link)
If not:
I did something similar recently and you will need to somehow turn your logs into a stream. I'd advise using the information found here (non-external link)
OutputStream Stream;
#GetMapping("/stream-sse-mvc")
public SseEmitter streamSseMvc() {
SseEmitter emitter = new SseEmitter();
ExecutorService sseMvcExecutor = Executors.newSingleThreadExecutor();
sseMvcExecutor.execute(() -> {
try {
Stream.map(sequence -> SseEmitter.event()
.id(""))
.event("EVENT_TYPE")
.data("String.valueOf(sequence)
.build());
emitter.send(event);
Thread.sleep(1000); //This does not need to be here
} catch (Exception ex) {
emitter.completeWithError(ex);
}
});
return emitter;
}
There might be better ways to reach your endpoints but without knowing what Frameworks you are using this is hard to answer. Essentially what we are doing is capturing all log output to a stream which is then broadcasted by an SSE.
Related
Can some one help how to configure NATS jet stream subscription in spring boot asynchronously example: looking for an equivalent annotation like #kafkalistener for Nats jetstream
I am able to pull the messages using endpoint but however when tried to pull messages using pushSubscription dispatcherhandler is not invoked. Need to know how to make the listener to be active and consume messages immediately once the messages are published to the subject.
Any insights /examples regarding this will be helpful, thanks in advance.
I don't know what is your JetStream retention policy, neither the way you want to subscribe. But I have sample code for WorkQueuePolicy push subscription, wish this will help you.
public static void subscribe(String streamName, String subjectKey,
String queueName, IMessageHandler iMessageHandler) throws IOException,
InterruptedException, JetStreamApiException {
long s = System.currentTimeMillis();
Connection nc = Nats.connect(options);
long e = System.currentTimeMillis();
logger.info("Nats Connect in " + (e - s) + " ms");
JetStream js = nc.jetStream();
Dispatcher disp = nc.createDispatcher();
MessageHandler handler = (msg) -> {
try {
iMessageHandler.onMessageReceived(msg);
} catch (Exception exc) {
msg.nak();
}
};
ConsumerConfiguration cc = ConsumerConfiguration.builder()
.durable(queueName)
.deliverGroup(queueName)
.maxDeliver(3)
.ackWait(Duration.ofMinutes(2))
.build();
PushSubscribeOptions so = PushSubscribeOptions.builder()
.stream(streamName)
.configuration(cc)
.build();
js.subscribe(subjectKey, disp, handler, false, so);
System.out.println("NatsUtil: " + durableName + "subscribe");
}
IMessageHandler is my custom interface to handle nats.io received messages.
First, configure the NATS connection. Here you will specify all your connection details like server address(es), authentication options, connection-level callbacks etc.
Connection natsConnection = Nats.connect(
new Options.Builder()
.server("nats://localhost:4222")
.connectionListener((connection, eventType) -> {})
.errorListener(new ErrorListener(){})
.build());
Then construct a JetStream instance
JetStream jetStream = natsConnection.jetStream();
Now you can subscribe to subjects. Note that JetStream consumers can be durable or ephemeral, can work according to push or pull logic. Please refer to NATS documentation (https://docs.nats.io/nats-concepts/jetstream/consumers) to make the appropriate choice for your specific use case. The following example constructs a durable push consumer:
//Subscribe to a subject.
String subject = "my-subject";
//queues are analogous to Kafka consumer groups, i.e. consumers belonging
//to the same queue (or, better to say, reading the same queue) will get
//only one instance of each message from the corresponding subject
//and only one of those consumers will be chosen to process the message
String queueName = "my-queue";
//Choosing delivery policy is analogous to setting the current offset
//in a partition for a consumer or consumer group in Kafka.
DeliverPolicy deliverPolicy = DeliverPolicy.New;
PushSubscribeOptions subscribeOptions = ConsumerConfiguration.builder()
.durable(queueName)
.deliverGroup(queueName)
.deliverPolicy(deliverPolicy)
.buildPushSubscribeOptions();
Subscription subscription = jetStream.subscribe(
subject,
queueName,
natsConnection.createDispatcher(),
natsMessage -> {
//This callback will be called for incoming messages
//asynchronously. Every subscription configured this
//way will be backed by its own thread, that will be
//used to call this callback.
},
true, //true if you want received messages to be acknowledged
//automatically, otherwise you will have to call
//natsMessage.ack() manually in the above callback function
subscribeOptions);
As for the declarative API (i.e. some form of #NatsListener annotation analogous to #KafkaListener from Spring for Apache Kafka project), there is none available out of the box in Spring. If you feel like you absolutely need it, you can write one yourself, if you are familiar with Spring BeanPostProcessor-s or other extension mechanism that can help to do that. Alternatively you can refer to 3rd party libs, it looks like a bunch of people (including myself) felt a bit uncomfortable when switching from Kafka to NATS, so they tried to bring the usual way of doing things with them from the Kafka world. Some examples can be found on github:
https://github.com/linux-china/nats-spring-boot-starter,
https://github.com/dstrelec/nats
https://github.com/amalnev/declarative-nats-listeners
There may be others.
I'm using spring webClient for call some Rest API.
try {
webClient.post().uri("apiUrl")
.bodyValue(someParams)
.retrive()
.awaitBody<SomeResponseClass>()
} catch (ex: WebCleintResponseException) {
ex.reponseBodyAsString
}
This is work right on under spring 5.3.12.
I can take response body in ex.
but over spring 5.3.13. is not work.
resposeBody is empty.
webClient.post().uri("apiUrl")
.bodyValue(someParams)
.awaitExchange { response: ClientResponse ->
if (response.satusCode().is2xxSuccessful)
response.awatieBody<SomeResponseClass>
else if (response.statusCodes().isError)
response.awatieBody<SomeResponseClass>
else throw response.createExceptionAndAwaite
}
This working one over spring 5.3.13.
So...What is different from this? I don't find anything on release note.
https://github.com/spring-projects/spring-framework/releases/tag/v5.3.13
I want to know what is changed.
Thanks guys.
Updated
I found this commit on github.
https://github.com/spring-projects/spring-framework/commit/9197f15a306a497b3aa10c6ed48c581b3e938f58
I want to be able to extract the List<Payload> from the Mono<List<Payload>> to pass it to a downstream service for processing (or maybe return from the read(RequestParams params) method, instead of it returning void):
#PostMapping("/subset")
public void read(#RequestBody RequestParams params){
Mono<List<Payload>> result = reader.read(params.getDate(), params.getAssetClasses(), params.getFirmAccounts(), params.getUserId(), params.getPassword());
....
}
where reader.read(...) is a method on an autowired Spring service utilizing a webClient to get the data from external web service API:
public Mono<List<Payload>> read(String date, String assetClasses, String firmAccounts, String id, String password) {
Flux<Payload> nodes = client
.get()
.uri(uriBuilder -> uriBuilder
.path("/api/subset")
.queryParam("payloads", true)
.queryParam("date", date)
.queryParam("assetClasses", assetClasses)
.queryParam("firmAccounts", firmAccounts)
.build())
.headers(header -> header.setBasicAuth("abc123", "XXXXXXX"))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, response -> {
System.out.println("4xx error");
return Mono.error(new RuntimeException("4xx"));
})
.onStatus(HttpStatus::is5xxServerError, response -> {
System.out.println("5xx error");
return Mono.error(new RuntimeException("5xx"));
})
.bodyToFlux(Payload.class);
Mono<List<Payload>> records = nodes
.collectList();
return records;
}
Doing a blocking result.block() is not allowed in WebFlux and throws an exception:
new IllegalStateException("block()/blockFirst()/blockLast() are blocking, which is not supported in thread ..." ;
What is a proper way to extract the contents of a Mono in WebFlux?
Is it some kind of a subscribe()? What would be the syntax?
Thank you in advance.
There is no "proper way" and that is the entire point. To get the value you need to block, and blocking is bad in webflux for many reasons (that I won't go into right now).
What you should do is to return the publisher all the way out to the calling client.
One of the things that many usually have a hard time understanding is that webflux works with a producer (Mono or Flux) and a subscriber.
Your entire service is also a producer, and the calling client can be seen as the subscriber.
Think of it as a long chain, that starts at the datasource, and ends up in the client showing the data.
A simple rule of thumb is that whomever is the final consumer of the data is the subscriber, everyone else is a producer.
So in your case, you just return the Mono<List<T> out to the calling client.
#PostMapping("/subset")
public Mono<List<Payload>> read(#RequestBody RequestParams params){
Mono<List<Payload>> result = reader.read(params.getDate(), params.getAssetClasses(), params.getFirmAccounts(), params.getUserId(), params.getPassword());
return result;
}
While the following does return the value of the Mono observable in the logs:
#PostMapping("/subset")
#ResponseBody
public Mono<ResponseEntity<List<Payload>>> read1(#RequestBody RequestParams params){
Mono<List<Payload>> result = reader.read(params.getDate(), params.getAssetClasses(), params.getFirmAccounts(), params.getUserId(), params.getPassword());
return result
.map(e -> new ResponseEntity<List<PayloadByStandardBasis>>(e, HttpStatus.OK));
}
the understanding I was seeking was a proper way to compose a chain of calls, with WebFlux, whereby a response from one of its operators/legs (materialized as as a result from a webclient call, producing a set of records, as above) could be passed downstream to another operator/leg to facilitate a side effect of saving those records in a DB, or something to that effect.
It would probably be a good idea to model each of those steps as a separate REST endpoint, and then have another endpoint for a composition operation which internally calls each independent endpoint in the right order, or would other design choices be more preferred?
That is ultimately the understanding I was looking for, so if anyone wants to share an example code as well as opinions to better implement the set of steps described above, I'm willing to accept the most comprehensive answer.
Thank you.
I am using web client in a spring application
I am facing memory leak issues while doing the same
I am using below code to get the response body for non 2XX response from service:
return client.get()
.uri(uriString)
.headers(ServiceCommonUtil.getHttpHeaderConsumer(headersMap))
.exchange()
.flatMap(clientResponse -> {
try {
clientResponse.body((clientHttpResponse, context) ->
clientHttpResponse.getBody());
logResponseStatus(clientResponse.statusCode(), serviceName);
return clientResponse.bodyToMono(String.class);
} catch (Exception e) {
return null;
}
})
and later on subscriber uses subscribe/ error block to process this response.
responseMono.subscribe(response -> {
//process response string
},error->{
//process error response
});
My question is, if i use dispose method on responseMono, it takes way long time for processing while without it i face memory leak issues.
Am i doing anything wrong here?
Yes, actually you are not consumming response in case of Exception is thrown.
If you want to use exchange() your responsibillity is to consume response.
See: docs
Take a look on toBodilessEntity()/ releaseBody() in 'ClientResponse` api.
Seems you've gotten a little complicated. Why a try/catch block in the clientResponse lambda? If your logResponseStatus throws a checked exception then handle it there. I suggest starting simpler.
Ex 1:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").exchange().flatMap(clientResponse->clientResponse.bodyToMono(String.class));
stringMono.subscribe(System.out::println);
Ex 2:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").exchange().flatMap(clientResponse->clientResponse.body(BodyExtractors.toMono(String.class)));
stringMono.subscribe(System.out::println);
Ex 3:
Mono<String> stringMono = webClient.get().uri("test").header("head", "value").retrieve().bodyToMono(String.class);
stringMono.subscribe(System.out::println);
For logging it is better to use ExchangeFilterFunctions. See How to intercept a request when using SpringBoot WebClient
.
I have one local spring boot application which is hitting another PCF deployed application. The PCF application is giving me list of Student.
I am hitting the same using:
Observable<Response> result = RxObservable.newClient()
.target(url)
.request()
.rx()
.get();
Now I am not able to get my List<Student> back from result Observable.
Tried lots of approaches but nothing really working i.e. subscribing to result etc etc.
So after struggling for few hours below is the solution for now.
Observable observable = result.map(response -> response.readEntity(List.class));
DeferredResult<ResponseEntity<Response>> deferredResult = new DeferredResult(Duration.ofMillis(10000L).toMillis());
observable.subscribe((response) -> {
ResponseEntity<Response> responseData = new ResonseEntity(response, HttpStatus.OK);
deferredResult.setResult(responseData);
}, deferredResult::setErrorResult);
Please suggest, if it can be improved.
Thanks