I am Working on Spring boot application and have to call client rest service there could see 10000 records.But i need to call their service using rest with limit of 50 records at time and next 50 records and so on up to 10000 records.Kindly please share your input on this.
It depends with how you want to do the limit. If you're using native Queries you can limit in the query. However this approach is not that smart because you'd need the pther data later. The smart idea is to limit in the view using approaches like pagination which are available in nearly all frontend or templating engines In case of spring thymeleaf together with technologies like bootstrap support pagination to limit results and view level rather than at api level
Related
I have Spring Boot application 2.5.7 where I set up a micrometer to scrape metrics
runtimeOnly("io.micrometer:micrometer-registry-prometheus")
When I make a request locally http://localhost:8081/actuator/prometheus
There are no performance problems with my application
But when I make a request to the actuator on the server with a high load
https://myserver:8081/actuator/prometheus
it returns a lot more data in response and it also slows down all request that is currently running on my server.
The problem appears even after one request to /actuator/prometheus
Is there any way to optimize the micrometer work(while returning the same ammount of metrics), so it will not slow down my application?
Without sufficient data it is hard to give a recommendation. If the slowness is due to insufficient memory/garbage collection, try increasing the memory of your application.
Reviewing the metrics being returned may also give you some ideas, for example if you have a high thread count, I think there is a pause when Micrometer iterates over the thread statuses. You could look into disabling that metric.
I have to improve performance of spring boot app, which is quite classical rest API + hibernate + postgres. we have 250k active users and want to extract some requests to be on slave balanced instances, and probably cache some data. For now i have only suspect that some requests need to be cached, But i want to make some audit and report, that some request called so many times that we should use other strategy, or "this" sql request fired every rest call so it's eat a lot DB lifetime which could be worked out using cache. Is there any best practice to make this kind of audit/analytic? Request-rate, request rate per user, SQL rate per request, SQL rate per user per request, and some other metrics
Spring Boot's metrics should give you a good starting point. The Spring MVC metrics should allow you to identify if there are certain types of request that are taking longer than others. Depending on how you are accessing your database, there are also DataSource metrics, Hibernate metrics, and Spring Data Repository metrics (new in Spring Boot 2.5) that may be of interest.
These metrics will be for your application as a whole rather than per-user. With over 250k active users, tagging metrics on a per-user basis almost certainly won't be practical. Unless you suspect that there are specific users that are problematic, I would at least start with the application-wide metrics and see how things go.
I have a use case/situation wherein, SQS(standard) will be flooded with messages (north of 500k+), a microservice (spring boot based) listens to these events, consumes it, and makes a rest API call (batch-based) to 3rd party SaaS system (have attached a high-level diagram for the same)
The limitation here is that the spring boot consumer can receive a max of 10 messages from the SQS, transform the payload, and makes the rest API call with these 10 messages(records).
Is there a way to aggregate these messages to say 100 messages, before making the rest API call (assuming that the target SaaS System accepts 100 records of data)? Would spring batch helps in this case?
Should I have to look at a different stack for this kind of need? Any help/guidance is much appreciated.
Thanks
What you are describing is actually the chunk-oriented processing model of Spring Batch: items could be read from the queue, accumulated in chunks of 100 items (that is the configurable chunk-size) and posted to your REST API in bulk mode.
Spring Batch handles the chunking of items (and much more) for you. So yes, even though I'm biased, I believe Spring Batch is a very good option for your use case.
Maybe you should try Spring Aggregator(Spring Integration).
The Aggregator combines a group of related messages, by correlating
and storing them until the group is deemed to be complete. At that
point, the aggregator creates a single message by processing the whole
group and sends the aggregated message as output.
https://docs.spring.io/spring-integration/reference/html/aggregator.html
And please refer to this GitHub repo for spring integration with AWS services
https://github.com/spring-projects/spring-integration-aws/tree/main/src/test/java/org/springframework/integration/aws
I'm assuming you are having multiple instances of your application and can scale up easily if required (since you have 500k+ messages). But still, your application is prone to data loss. So building a reliable system is always challenging. Since you are already on the cloud and maybe you should think about utilizing different cloud services.
I think for your case, you should have a look at the AWS Kinesis dataStream and Kinesis data fire hose.
You can refer this,
https://aws.amazon.com/blogs/big-data/stream-data-to-an-http-endpoint-with-amazon-kinesis-data-firehose/
I have rest API through which I am sending and getting massages to the Kafka server using spring-boot. Now I want to save those messages to elasticsearch. How to do it can anyone help?
Actually this is a systematic job in which case, is somehow like setting up a database storage architecture.
TO BE SIMPLE AND SHORT:
First you need to decide which ES version you want to use, because there are some breaking changes between ES 2.x to 7.x. And those differences may affect the way you design the schema of your storage.
Assume you use latest 7.x ES, you will need to create index(es) where you want the data fetched from kafka to be stored into. Checkout https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
Later you have indexes created, you need to apply and learn some basic knowledge about ES high level rest client and low level rest client. The low level rest client enables you the basic connection to ES cluster via HTTP. And high level rest client apis give you sufficient ways to do ops like documents CRUD, search, aggregations for your data. You can easily find dependencies via maven and use them in your Spring Boot Application. Checkout https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-high.html
I am in a dilemma over to use spring's rest template or elasticsearch's own high/low rest client while searching in es . Does es client provide any advantage like HTTP connection pooling , performance while compared to spring rest template . Which of the two take less time in getting response from the server . Can some one please explain this ?
The biggest advantage of using Spring Data Elasticsearch is that you don't have to bother about the things like converting your requests/request bodies/responses from your POJO domain classes to and from the JSON needed by Elasticsearch. You just use the methods defined in the ElasticsearchOperations class which is implemented by the *Template classes.
Or going one abstraction layer up, use the Repository interfaces the all the Spring Data modules provide to store and search/retrieve your data.
Firstly, This is a very broad question. Not sure if it suits the SO guidelines.
But my two cents:
High Level Client uses Low Level client which does provide connection pooling
High Level client manages the marshalling and unmarshalling of the Elastisearch query body and response, so it might be easier to work using the APIs.
On the other hand, if you are familiar with the Elasticsearch querying by providing the JSON body then you might find it a bit difficult to translate between the JSON body and the Java classes used for creating the query (i.e when you are using Kibana console or other REST API tools)
I generally overcome this by logging the query generated by the Java API so that I can use it with Kibana console or other REST API tools.
Regarding which one is efficient- the library will not matter that much to affect the response times.
If you want to use Spring Reactive features and make use of WebClient, ES Libraries do provide support for Async search.
Update:
Please check the answer by Wim Van den Brande below. He has mentioned a very valid point of using Transport Client which has been deprecated over REST API.
So it would be interesting to see how RestTemplate or Spring Data ElasticSearch will update their API to replace TransportClient.
One important remark and caveat with regards to the usage of Spring Data Elasticsearch. Currently, Spring Data Elasticsearch doesn't support the communication by the High Level REST Client API. They are using the transport client. Please note, the TransportClient is deprecated as of Elasticsearch 7.0.0 and is expected to be removed in Elasticsearch 8.0!!!
FYI, this statement has been confirmed already by another post: Elasticsearch Rest Client with Spring Data Elasticsearch