Spring Integration Java DSL: How to loop the paged Rest service? - spring

Question

How to loop the paged Rest service with the Java DSL Http.outboundGatewaymethod?
The rest URL is for example
http://localhost:8080/people?page=3
and it returns for example
"content": [
{"name": "Mike",
"city": "MyCity"
},
{"name": "Peter",
"city": "MyCity"
},
...
]
"pageable": {
"sort": {
"sorted": false,
"unsorted": true
},
"pageSize": 20,
"pageNumber": 3,
"offset": 60,
"paged": true,
"unpaged": false
},
"last": false,
"totalElements": 250,
"totalPages": 13,
"first": false,
"sort": {
"sorted": false,
"unsorted": true
},
"number": 3,
"numberOfElements": 20,
"size": 20
}
where the variable totalPages tells the total pages amount.
So if the implementation
integrationFlowBuilder
.handle(Http
.outboundGateway("http://localhost:8080/people?page=3")
.httpMethod(HttpMethod.GET)
.expectedResponseType(String.class))
access one page, how to loop all the pages?

Answer 1

The easiest way to do this is like wrapping the call to this Http.outboundGateway() with the #MessagingGateway and provide a page number as an argument:
#MessagingGateway
public interface HttpPagingGateway {
#Gateway(requestChannel = "httpPagingGatewayChannel")
String getPage(int page);
}
Then you get a JSON as a result, where you can convert it into some domain model or just perform a JsonPathUtils.evaluate() (based on json-path) to get the value of the last attribute to be sure that you need to call that getPage() for the page++ or not.
The page argument is going to be a payload of the message to send and that can be used as an uriVariable:
.handle(Http
.outboundGateway("http://localhost:8080/people?page={page}")
.httpMethod(HttpMethod.GET)
.uriVariable("page", Message::getPayload)
.expectedResponseType(String.class))
Of course, we can do something similar with Spring Integration, but there are going to be involved filter, router and some other components.
UPDATE
First of all I would suggest you to create a domain model (some Java Bean), let's say PersonPageResult, to represent that JSON response and this type to the expectedResponseType(PersonPageResult.class) property of the Http.outboundGateway(). The RestTemplate together with the MappingJackson2HttpMessageConverter out-of-the-box will do the trick for you to return such an object as a reply for the downstream processing.
Then, as I said before, looping would be better done from some Java code, which you could wrap to the service activator call. For this purpose you should daclare a gateway like this:
public interface HttpPagingGateway {
PersonPageResult getPage(int page);
}
Pay attention: no annotations at all. The trick is done via IntegrationFlow:
#Bean
public IntegrationFlow httpGatewayFlow() {
return IntegrationFlows.from(HttpPagingGateway.class)
.handle(Http
.outboundGateway("http://localhost:8080/people?page={page}")
.httpMethod(HttpMethod.GET)
.uriVariable("page", Message::getPayload)
.expectedResponseType(PersonPageResult.class))
}
See IntegrationFlows.from(Class<?> aClass) JavaDocs.
Such a HttpPagingGateway can be injected into some service with hard looping logic:
int page = 1;
boolean last = false;
while(!last) {
PersonPageResult result = this.httpPagingGateway.getPage(page++);
last = result.getLast();
List<Person> persons = result.getPersons();
// Process persons
}
For processing those persons I would suggest to have separate IntegrationFlow, which may start from the gateway as well or you can just send a Message<List<Person>> to its input channel.
This way you will separate concerns about paging and processing and will have a simple loop logic in some POJO method.