I'm working a project with tens of services, using Spring sleuth and zipkin, but I was wondering if there is any way to conditionally propagate logs to zipkin server.
Actually, would be perfect if the log was propagated only when The distributed transaction failed, (like using a saga pattern). The case is, we have a huge workload (millions of request per hour) and we are interest only in failed request.
You can't propagate logs to Zipkin, you can publish Spans.
Depending on your needs, you can use a SamplerFunction, a Sampler or a SpanHandler, see this answer: https://stackoverflow.com/a/69981877/971735
Related
Spring + Apache Kafka noob here. I'm wondering if its advisable to run a single Spring Boot application that handles both producing messages as well as consuming messages.
A lot of the applications I've seen using Kafka lately usually have one separate application send/emit the message to a Kafka topic, and another one that consumes/processes the message from that topic. For larger applications, I can see a case for separate producer and consumer applications, but what about smaller ones?
For example: I'm a simple app that processes HTTP requests => send requests to a third party service, but to ensure retryability, I put the request on a Kafka queue with a service using the #Retryable annotation?
And what other considerations might come into play since it would be on the Spring framework?
Note: As your question states, what'll say is more of an advice based on my beliefs and experience rather than some absolute truth written in stone.
Your use case seems more like a proxy than an actual application with business logic. You should make sure that making this an asynchronous service makes sense - maybe it's good enough to simply hold the connection until you get a response from the 3p, and let your client handle retries if you get an error - of course, you can also retry until some timeout.
This would avoid common asynchronous issues such as making your client need to poll or have a webhook in order to get a result, or making sure a record still makes sense to be processed after a lot of time has elapsed after an outage or a high consumer lag.
If your client doesn't care about the result as long as it gets done, and you don't expect high-throughput on either side, a single Spring Boot application should be enough for handling both producer and consumer sides - while also keeping it simple.
If you do expect high throughput, I'd look into building a WebFlux based application with the reactor-kafka library - high throughput proxies are an excellent use case for reactive applications.
Another option would be having a simple serverless function that handles the http requests and produces the records, and a standard Spring Boot application to consume them.
TBH, I don't see a use case where having two full-fledged java applications to handle a proxy duty would pay off, unless maybe you have a really sound infrastructure to easily manage them that it doesn't make a difference having two applications instead of one and using more resources is not an issue.
Actually, if you expect really high traffic and a serverless function wouldn't work, or maybe you want to stick to Java-based solutions, then you could have a simple WebFlux-based application to handle the http requests and send the messages, and a standard Spring Boot or another WebFlux application to handle consumption. This way you'd be able to scale up the former in order to accommodate the high traffic, and independently scale the later in correspondence with your performance requirements.
As for the retry part, if you stick to non-reactive Spring Kafka applications, you might want to look into the non-blocking retries feature from Spring Kafka. This will enable your consumer application to process other records while waiting to retry a failed one - the #Retryable approach is deprecated in favor of DefaultErrorHandler and both will block consumption while waiting.
Note that with that you lose ordering guarantees, so use it only if the order the requests are processed is not important.
I have to improve performance of spring boot app, which is quite classical rest API + hibernate + postgres. we have 250k active users and want to extract some requests to be on slave balanced instances, and probably cache some data. For now i have only suspect that some requests need to be cached, But i want to make some audit and report, that some request called so many times that we should use other strategy, or "this" sql request fired every rest call so it's eat a lot DB lifetime which could be worked out using cache. Is there any best practice to make this kind of audit/analytic? Request-rate, request rate per user, SQL rate per request, SQL rate per user per request, and some other metrics
Spring Boot's metrics should give you a good starting point. The Spring MVC metrics should allow you to identify if there are certain types of request that are taking longer than others. Depending on how you are accessing your database, there are also DataSource metrics, Hibernate metrics, and Spring Data Repository metrics (new in Spring Boot 2.5) that may be of interest.
These metrics will be for your application as a whole rather than per-user. With over 250k active users, tagging metrics on a per-user basis almost certainly won't be practical. Unless you suspect that there are specific users that are problematic, I would at least start with the application-wide metrics and see how things go.
I have a use case/situation wherein, SQS(standard) will be flooded with messages (north of 500k+), a microservice (spring boot based) listens to these events, consumes it, and makes a rest API call (batch-based) to 3rd party SaaS system (have attached a high-level diagram for the same)
The limitation here is that the spring boot consumer can receive a max of 10 messages from the SQS, transform the payload, and makes the rest API call with these 10 messages(records).
Is there a way to aggregate these messages to say 100 messages, before making the rest API call (assuming that the target SaaS System accepts 100 records of data)? Would spring batch helps in this case?
Should I have to look at a different stack for this kind of need? Any help/guidance is much appreciated.
Thanks
What you are describing is actually the chunk-oriented processing model of Spring Batch: items could be read from the queue, accumulated in chunks of 100 items (that is the configurable chunk-size) and posted to your REST API in bulk mode.
Spring Batch handles the chunking of items (and much more) for you. So yes, even though I'm biased, I believe Spring Batch is a very good option for your use case.
Maybe you should try Spring Aggregator(Spring Integration).
The Aggregator combines a group of related messages, by correlating
and storing them until the group is deemed to be complete. At that
point, the aggregator creates a single message by processing the whole
group and sends the aggregated message as output.
https://docs.spring.io/spring-integration/reference/html/aggregator.html
And please refer to this GitHub repo for spring integration with AWS services
https://github.com/spring-projects/spring-integration-aws/tree/main/src/test/java/org/springframework/integration/aws
I'm assuming you are having multiple instances of your application and can scale up easily if required (since you have 500k+ messages). But still, your application is prone to data loss. So building a reliable system is always challenging. Since you are already on the cloud and maybe you should think about utilizing different cloud services.
I think for your case, you should have a look at the AWS Kinesis dataStream and Kinesis data fire hose.
You can refer this,
https://aws.amazon.com/blogs/big-data/stream-data-to-an-http-endpoint-with-amazon-kinesis-data-firehose/
If a microservice is not responding due to any of the following reasons, how do we ensure the overall application availability?
Microservice crashes
Network partition happens or other transient error happens
Service is overloaded
other microservice calling the same microservice
If you have services calling one another, that doesn't sound like they are using Kafka, then.
If you have applications sending to Kafka, then those messages are persisted to the broker logs. Any downstream consumer can stay offline for as long as the messages are (configurably) retained in the Kafka cluster.
Ultimately, when using Kafka (any persistent message queue), services do not know about one another, and only the brokers.
You should avoid coupling in microservices architecture as much as possible.
In your case, I guess you are sending a read-only request to a microservice to get a data but called microservice is not up. So caller microservice can't do its job.
To avoid this kind of situations you can use data duplication technique. In this technique microservice which is the source of the data send insert, update, delete information about the data as an event with using a broker like Kafka. Then other microservices which also needs to this data get the data from corresponding topic. By this way, you don't need to make a read-only request to get the data. Then you will avoid coupling between microservices.
What will happen in that case?
In this case, if there is no redundancy for microservice which is called, caller microservice will get an exception like "No instances available for CalledMicroservice"
I am working on adding logging/monitoring functionality for multiple spring integration deployments. I want to create a logger transaction at the start of the workflow and close the log transaction at the end of the workflow. At the end of the logger transaction I will send out the logs and metrics to a centralized logging server. At the end of the day I want to see the logs for all the messages that went through workflows across multiple spring integration deployments.
The spring integration deployments are across a lot of teams and I cant count on each team to add the logging code for me, so I want to write code that will run across the spring integration deployments.
To start a log transaction,the solution was to use a global-channel interceptor on a set of inbound messaging channels. All the workflows built across deployments use the same set of inbound channels, so the start log transaction interceptor will run.
Also I pass the logger transaction details as part of the message headers
But I am having trouble figuring out a solution for ending the transaction. The workflows can be synchronous as well as asynchronous. Also not all endpoints within the workflow will have a output channel.
Any strategies/ideas on how can I close the logger transaction ?
An example of a sample workflow is shown in the image below:
When you have a flow that ends with a channel adapter (or other endpoint that produces no result), make the last channel a publish-subscribe-channel; subscribe a second endpoint to that channel that terminates the "transaction".
I generally prefer to add an order attribute on such endpoints - make the regular endpoint order="1" and the flow terminator order="2" - clearly indicating it is called after the main endpoint.
It is important that no task executor is added to the channel so the endpoints are called serially on the same thread.