I need to develop an application to perform the following activities using Spring Batch. Read data from the database --> Process data and prepare rest API request --> Write or Post into third party restful service.
I have seen a lot of examples for reading from the database to write to CSV, DB, JMS.
But I don't find any options to write into web service. Is it possible to perform this activity using spring batch or please suggest some other technology.
Spring Batch does not have a RestItemReader since the possibilities are a bit too great as to what we'd need to send. That being said, writing your own ItemWriter implementation that uses a RestTemplate to send the data would be very straight forward to do.
Related
I have a small Java SE application, it´s practically a fat client sitting on top of a database. To further my Java skills, I decided to make a client-server application out of it. The server application communicates with the database and handles all kinds of lengthy operations, while the client application only receives the results, mostly ArrayLists of moderate length and primitives.
To this end, I started reading on RMI and did the Oracle tutorial, which I found surprisingly difficult to understand and even get to work.
Is there anything else I can use instead of RMI, without having to dive into JavaEE?
One way I would suggest would be to use JSON as the format for data exchange. You can use GSON to convert the data from Java objects to JSON and back. The transport could be done directly on the HTTP protocol using REST. You can use Jersey as a REST server/client or roll your own (since you don't want to use JEE, which Jersey is part of).
SIMON is as simple to use as RMI, but with fewer traps in the initial setup. It also has some advantages over RMI. Here is a simple hello-world example:
http://dev.root1.de/projects/simon/wiki/Sample_helloworld110
I take it RMI = Remote Method Invocation ...
You can have a look at XMLRPC, JSONRPC, JMS, or if you want to roll your own, use JSON to POST messages and convert the JSON back to a java object on the other side using GSON (from Google) or setup RabbitMQ and use AMQP to submit and receive messages if you don't want to handle the POSTing yourself, Spring AMQP makes it fairly easy to do this.
I am using Spring Batch ItemReader to read data from a database and then using FlatFileItemWriter to write the data to a file.
Once the data is written to a file, I need this specific file to be transferred to an FTP server. Can I do this via Spring Batch or should I use Spring Integration? Could you please also provide an example?
There is nothing wrong in combining both frameworks. They really work together smoothly.
Since you say that you already have a Spring Batch ItemReader and your OK writing into a file using FlatFileItemWriter, so you are have way out for your whole solution.
In the end you can use a Providing Feedback with Informational Messages to get a file and send it to FTP using Spring Integration FTP Outbound Channel adapter.
See more info in Spring Batch Docs: https://docs.spring.io/spring-batch/4.2.x/reference/html/spring-batch-integration.html#providing-feedback-with-informational-messages
And in Spring Integration about FTP support: https://docs.spring.io/spring-integration/docs/current/reference/html/#ftp
I am curious to learn, Which design approach you would take in following scenario?
I have a FTP server where my java spring app (Service 1) will go and fetch the file. I will then store that file on to a S3 bucket. this file may have up to a million records. There is going to be another micro service (Service 2) which would need to need read this file data.
How would you design and expose these million records from service1 so that those would readable by Service 2?
Which technology stack you would use on either side and why?
these both services are going to hosted on same servers(For now).
Can you suggest a quick and efficient solution?
Thanks
Here's my approach. I wouldn't even create 2 services provided your scope of work. The only service will use Spring Integration and Spring Batch.
Spring Integration part will get the file from FTP and copy it to S3 and hands over to Spring Batch for processing
After processing with Batch, save the records.
If the job is to run once in a while and don't have much records, I would even not use Spring Batch. I would just use Spring Integration and process the records in parallel. But if you know you'll have millions of records and tons of files, you should consider using Spring Batch for batch processing.
I have requirement something like this:
once the request is received by my service, i need to send it 2-3 third party servers at a time and get the response from all server and return the response.
How can I achieve that?.
My thought : I can create separate threads for different servers and send the request to all servers parallely, but here the issue is, how I will come to know the threads are finished and consolidate the response from all servers and return to caller.
Is there any other way to do in spring boot(micro service)?.
You can leverage asyn feature supported by Spring Framework. Let's see the folllowing example which issue multiple calls in Async style from Spring's official guides:Async Method
Another possible solution for internal communications between Microserives is to use the message queue.
You didn't specify what type of services are you consuming. If they are HTTP, you may want to use some Enterprise Integrations abstractions (most popular are Spring Integration and Apache Camel).
If you don't want to introduce message bus solution into your microservice, you may want to take a look at AsyncRestTemplate
thanks for attention,i defined a combine spring batch and spring integration project and communicate with ftp server to retrieve file and process on it and write on ftp, i am looking for a good architecture for my project, i designed an architecture with spring integration as bellow diagram:
when retrieve file from server process on it and route files based on condition to mvChannel and toGet channel, i have many process scenario on the retrieved file from server that i defined a router that router handle the scenario , and route to job channels and run spring batch
now, my is question is that are right the architecture?
Really looks good. It is is typical pattern to get gain from both Spring Integration and Spring Batch worlds integration them.
However I don't see reason in the last <aggregator>. All your jobs can send their result to the <int-ftp:outbound-gateway command="PUT">. Looks like no need to wait for all results to do some analysis on them in the single place.