I own a Spring boot Rest API which is consumed by many external Applications. This API receives a List of documents(base64 String) in one request. Each document is of around 100MB and most of the time there are 6,7 documents in Request payload. Let's say if 7 documents are in list it will consume 700MB in memory which is too much. I want to make it memory efficient.
I Can't ask Consumer of API to send document one by one, I have to receive them at once in a request but i want load documents one by one in memory instead of loading at once.
Here is example.
#PostMapping("/documents")
Employee newEmployee(#RequestBody List<String> DocuemntsInBase64) {
}
Also let me know if JsonSurfer can help in rescue.
Hey i have solve this mystery simply by using HttpServletRequest and JsonSurfer Library.
Here a sample code:
#RequestMapping(value = "/postDocuments",
method = RequestMethod.POST,
consumes = {"application/json", "application/xml"},
produces = {"application/json", "application/xml"})
#ResponseStatus(HttpStatus.CREATED)
public void createHotel(HttpServletRequest request, HttpServletResponse response) {
JsonSurfer surfer = JsonSurferJackson.INSTANCE;
try {
surfer.configBuilder()
.bind("Your JsonPath of list", new JsonPathListener() {
#Override
public void onValue(Object value, ParsingContext context) {
logger.info("Document of size: "+value.toString().getBytes().length);
}
})
.buildAndSurf(request.getReader());
} catch (IOException e) {
e.printStackTrace();
}
}
Related
The spring integration flow I wrote has to get files (some of them are as big as 4G) from a rest service and transfer them to a remote shared drive. For downloading them from the rest service I configured this simple component:
#Bean
public HttpRequestExecutingMessagehandler httpDownloader (RestTemplate template){
Expression expr = (new SpelExpressionParser()).parseExpression("payload.url");
HttpRequestExecutingMessagehandler handler = new HttpRequestExecutingMessagehandler (expr, template);
handler.setExpectedResponseType(byte[].class);
handler.setHttpMethod(GET);
return handler;
}
Unfortunately this won't scale meaning for larger files it will eventually throw java.lang.OutOfMemoryError: Java heap space, even if i add more memory with -Xmx or -XXMaxPermSize
So my question is, what to do in order to avoid these problems no matter how big the files will be?
I think I have answered you in some other similar your question that Spring RestTemplate is not designed for streaming response body. It is explained in this SO thread: Getting InputStream with RestTemplate.
One of the solution which may work for your is to write a custom HttpMessageConverter which would return a File object containing data from HTTP response. This article explains how to do that with the ResponseExtractor, but something like FileHttpMessageConverter is not so hard to implement based on experience from that article. See StreamUtils.copy(InputStream in, OutputStream out)
Then you inject this FileHttpMessageConverter into your HttpRequestExecutingMessagehandler - setMessageConverters(List<HttpMessageConverter<?>> messageConverters).
Your service for remote shared drive should already deal with this local temporary file to get that large content without consuming memory.
See also this one about possible approach via WebFlux: https://www.amitph.com/spring-webclient-large-file-download/
Created this starting from ByteArrayHttpMessageConverter class and injected it into the custom RestTemplate I use. But this solution is based on using a File message, which is not quite the streaming I was hoping for.
public class FileCustomConverter extends AbstractHttpMessageConverter<File> {
public FileCustomConverter() {
super(new MediaType[]{MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL});
}
public boolean supports(Class<?> clazz) {
return File.class == clazz;
}
public File readInternal(Class<? extends File> clazz, HttpInputMessage inputMessage) throws IOException {
File outputFile = File.createTempFile(UUID.randomUUID().toString(), ".tmp");
OutputStream outputStream = new FileOutputStream(outputFile);
StreamUtils.copy(inputMessage.getBody(), outputStream);
outputStream.close();
return outputFile;
}
protected Long getContentLength(File bytes, #Nullable MediaType contentType) {
return bytes.length();
}
protected void writeInternal(File file, HttpOutputMessage outputMessage) throws IOException {
InputStream inputStream = new FileInputStream(file);
StreamUtils.copy(inputStream, outputMessage.getBody());
inputStream.close();
}
}
I am calling a service in an orders controller which receives a multipart file and processes it and saving it into a database. I am trying to create a Spring Rest Doc for it but it is not even hitting the endpoint. I am creating a list of orders which is what the service expects. It receives the order as a stream as shown and converts into a stream of orders before saving it into a database. I have shown the main part of the controller and my code for generating the rest docs. When I run the code I get the following exception, it never even hits the endpoint when I set a breakpoint. I also used fileupload() but that did not work either.
Exception is:
Content type = application/json
Body = {"path":"/orders/order_reception","exceptionName":
"MissingServletRequestPartException","message":"Required request part 'uploadFile' is not
present",
"rootExceptionName":"MissingServletRequestPartException",
"rootMessage":"MissingServletRequestPartException: Required request part 'uploadFile' is not present"}
#RestController
#RequestMapping(value = "/orders")
#Validated
class OrderController{
#PostMapping(path = "/order_reception")
public ResponseEntity receiveData(#RequestPart MultipartFile uploadFile,
HttpServletRequest request,
HttpServletResponse response) {
if (!uploadFile.isEmpty()) {
try {
Reader reader = new InputStreamReader(request.getInputStream()));
... save file
return new ResponseEntity<>(HttpStatus.HttpStatus.CREATED);
} catch (Exception e) {
return new ResponseEntity(HttpStatus.INTERNAL_SERVER_ERROR);
}
}
return new ResponseEntity(HttpStatus.BAD_REQUEST);
}
#Test
public void sendData() throws Exception {
ObjectMapper mapper = new ObjectMapper();
Order order = repository.getOrder("1233333");
List<Order> orderList = new ArrayList<>():
resourceList.add(order);
MockMultipartFile orderFile = new MockMultipartFile("order-data", "order.json", "application/json",
mapper.writeValueAsString(orderList).getBytes(Charset.defaultCharset()));
mockMvc.perform(multipart("/orders/order_reception")
.file(orderFile))
.andExpect(status().isCreated())
.andDo(document("send-order",
preprocessRequest(prettyPrint()),
preprocessResponse(prettyPrint())));
}
Thank you Marten Deinum, your suggestion that the file name was wrong fixed it.
I simply changed name in the MockMultipartFile( "uploadsFile", ...)
I'm developing REST service which, in turn, will query slow legacy system so response time will be measured in seconds. We also expect massive load so I was thinking about asynchronous/non-blocking approaches to avoid hundreds of "servlet" threads blocked on calls to slow system.
As I see this can be implemented using AsyncContext which is present in new servlet API specs. I even developed small prototype and it seems to be working.
On the other hand it looks like I can achieve the same using Spring WebFlux.
Unfortunately I did not find any example where custom "backend" calls are wrapped with Mono/Flux. Most of the examples just reuse already-prepared reactive connectors, like ReactiveCassandraOperations.java, etc.
My data flow is the following:
JS client --> Spring RestController --> send request to Kafka topic --> read response from Kafka reply topic --> return data to client
Can I wrap Kafka steps into Mono/Flux and how to do this?
How my RestController method should look like?
Here is my simple implementation which achieves the same using Servlet 3.1 API
//took the idea from some Jetty examples
public class AsyncRestServlet extends HttpServlet {
...
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
String result = (String) req.getAttribute(RESULTS_ATTR);
if (result == null) { //data not ready yet: schedule async processing
final AsyncContext async = req.startAsync();
//generate some unique request ID
String uid = "req-" + String.valueOf(req.hashCode());
//share it to Kafka receive together with AsyncContext
//when Kafka receiver will get the response it will put it in Servlet request attribute and call async.dispatch()
//This doGet() method will be called again and it will send the response to client
receiver.rememberKey(uid, async);
//send request to Kafka
sender.send(uid, param);
//data is not ready yet so we are releasing Servlet thread
return;
}
//return result as html response
resp.setContentType("text/html");
PrintWriter out = resp.getWriter();
out.println(result);
out.close();
}
Here's a short example - Not the WebFlux client you probably had in mind, but at least it would enable you to utilize Flux and Mono for asynchronous processing, which I interpreted to be the point of your question. The web objects should work without additional configurations, but of course you will need to configure Kafka as the KafkaTemplate object will not work on its own.
#Bean // Using org.springframework.web.reactive.function.server.RouterFunction<ServerResponse>
public RouterFunction<ServerResponse> sendMessageToTopic(KafkaController kafkaController){
return RouterFunctions.route(RequestPredicates.POST("/endpoint"), kafkaController::sendMessage);
}
#Component
public class ResponseHandler {
public getServerResponse() {
return ServerResponse.ok().body(Mono.just(Status.SUCCESS), String.class);
}
}
#Component
public class KafkaController {
public Mono<ServerResponse> auditInvalidTransaction(ServerRequest request) {
return request.bodyToMono(TopicMsgMap.class)
// your HTTP call may not return immediately without this
.subscribeOn(Schedulers.single()) // for a single worker thread
.flatMap(topicMsgMap -> {
MyKafkaPublisher.sendMessages(topicMsgMap);
}.flatMap(responseHandler::getServerResponse);
}
}
#Data // model class just to easily convert the ServerRequest (from json, for ex.)
// + ~#constructors
public class TopicMsgMap() {
private Map<String, String> topicMsgMap;
}
#Service // Using org.springframework.kafka.core.KafkaTemplate<String, String>
public class MyKafkaPublisher {
#Autowired
private KafkaTemplate<String, String> template;
#Value("${topic1}")
private String topic1;
#Value("${topic2}")
private String topic2;
public void sendMessages(Map<String, String> topicMsgMap){
topicMsgMap.forEach((top, msg) -> {
if (topic.equals("topic1") kafkaTemplate.send(topic1, message);
if (topic.equals("topic2") kafkaTemplate.send(topic2, message);
});
}
}
Guessing this isn't the use-case you had in mind, but hope you find this general structure useful.
There is several approaches including KafkaReplyingRestTemplate for this problem but continuing your approach in servlet api's the solution will be something like this in spring Webflux.
Your Controller method looks like this:
#RequestMapping(path = "/completable-future", method = RequestMethod.POST)
Mono<Response> asyncTransaction(#RequestBody RequestDto requestDto, #RequestHeader Map<String, String> requestHeaders) {
String internalTransactionId = UUID.randomUUID().toString();
kafkaSender.send(Request.builder()
.transactionId(requestHeaders.get("transactionId"))
.internalTransactionId(internalTransactionId)
.sourceIban(requestDto.getSourceIban())
.destIban(requestDto.getDestIban())
.build());
CompletableFuture<Response> completableFuture = new CompletableFuture();
taskHolder.pushTask(completableFuture, internalTransactionId);
return Mono.fromFuture(completableFuture);
}
Your taskHolder component will be something like this:
#Component
public class TaskHolder {
private Map<String, CompletableFuture> taskHolder = new ConcurrentHashMap();
public void pushTask(CompletableFuture<Response> task, String transactionId) {
this.taskHolder.put(transactionId, task);
}
public Optional<CompletableFuture> remove(String transactionId) {
return Optional.ofNullable(this.taskHolder.remove(transactionId));
}
}
And finally your Kafka ResponseListener looks like this:
#Component
public class ResponseListener {
#Autowired
TaskHolder taskHolder;
#KafkaListener(topics = "reactive-response-topic", groupId = "test")
public void listen(Response response) {
taskHolder.remove(response.getInternalTransactionId()).orElse(
new CompletableFuture()).complete(response);
}
}
In this example I used internalTransactionId as CorrelationId but you can use "kafka_correlationId" that is a known kafka header.
In spring I have a controller with an endpoint like so:
#RequestMapping(method = RequestMethod.POST)
#ResponseStatus(HttpStatus.CREATED)
#ResponseBody
public OutputStuff createStuff(#RequestBody Stuff stuff) {
//my logic here
}
This way if doing a POST on this endpoint, the JSON in request body will be automatically deserialized to my model (Stuff). The problem is, I just got a requirement to log the raw JSON as it is coming in! I tried different approaches.
Inject HttpServletRequest into createStuff, read the body there and log:
Code:
#RequestMapping(method = RequestMethod.POST)
#ResponseStatus(HttpStatus.CREATED)
#ResponseBody
public OutputStuff createStuff(#RequestBody Stuff stuff, HttpServletRequest req) {
StringBuilder sb = new StringBuilder();
req.getReader().getLines().forEach(line -> {
sb.append(line);
});
//log sb.toString();
//my logic here
}
The problem with this is that by the time I execute this, the reader's InputStream would have already been executed to deserialize JSON into Stuff. So I will get an error because I can't read the same input stream twice.
Use custom HandlerInterceptorAdapter that would log raw JSON before the actual handler is called.
Code (part of it):
public class RawRequestLoggerInterceptor extends HandlerInterceptorAdapter {
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
StringBuilder sb = new StringBuilder();
req.getReader().getLines().forEach(line -> {
sb.append(line);
});
//log sb.toString();
return true;
}
}
The problem with this tho is, that by the time the deserialization to stuff happens, the InputStream from the request would have been read already! So I would get an exception again.
Another option I considered, but not implemented yet, would be somehow forcing Spring to use my custom implementation of HttpServletRequest that would cache the input stream and allow multiple read of it. I have no idea if this is doable tho and I can't find any documentation or examples of that!
Yet another option would be not to read Stuff on my endpoint, but rather read the request body as String, log it and then deserialize it to Stuff using ObjectMapper or something like that. I do not like this idea either tho.
Are there better solutions, that I did not mention and/or am not aware of? I would appreciate help. I am using the latest release of SpringBoot.
To read the request body multiple times, we must cache the initial payload. Because once the original InputStream is consumed we can't read it again.
Firstly, Spring MVC provides the ContentCachingRequestWrapper class which stores the original content. So we can retrieve the body multiple times calling the getContentAsByteArray() method.
So in your case, you can make use of this class in a Filter:
#Component
public class CachingRequestBodyFilter extends GenericFilterBean {
#Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest currentRequest = (HttpServletRequest) servletRequest;
ContentCachingRequestWrapper wrappedRequest = new ContentCachingRequestWrapper(currentRequest);
// Other details
chain.doFilter(wrappedRequest, servletResponse);
}
}
Alternatively, you can register CommonsRequestLoggingFilter in your application. This filter uses ContentCachingRequestWrapper behind the scenes and is designed for logging the requests.
As referenced in this post: How to Log HttpRequest and HttpResponse in a file?, spring provides the AbstractRequestLoggingFilter you can use to log the request.
AbstractRequestLoggingFilter API Docs, found here
I also tried to do that in Spring but i could not find way to pass my custom http request to chain so what did was,i have written traditional j2ee filter in that i have passed my custom http request to chain that is it then onward i can read http request more than once
Check this example http://www.myjavarecipes.com/how-to-read-post-request-data-twice-in-spring/
I'm building a small application to serve as a client for some third party library here at work. The API states that a Webhookis needed to respond some asynchronous events, but all their methods have the very same signature, apart from a changing _method field between the calls. For example, I have a _method = ping, media, etc.
I'd like to have separate methods on my controller to respond for each one of these methods. If the app allowed me to specify different URLs for each method it would be easy to use Spring MVC's #RequestMapping for each one of them. But I have to specify a single endpoint to receive all calls.
Is there a way (for example using Spring's HttpMessageConverter or something like that) to map different controller methods based on what the Request Body is? I've already tried with #RequestBody, #RequestParam but didn't seem to find anything.
I really, really didn't want to use a bunch of case, switch methods on a front controller to dispatch actions based on my _method field that comes with my POST data, so I happen to believe someone had this problem before and solved it intelligently.
Thanks a lot!
Edit 1: Providing source code
#Controller
#RequestMapping("/webhooks")
public class WebhookController {
#RequestMapping(method = RequestMethod.POST, params = {"_method=ping"})
#ResponseBody
public String ping(){
return "pong";
}
#RequestMapping(method = RequestMethod.POST, params = {"_method=media"})
#ResponseBody
public String media(){
return "media";
}
}
This is the answer:
{
"timestamp": 1440875190389,
"status": 400,
"error": "Bad Request",
"exception": "org.springframework.web.bind.UnsatisfiedServletRequestParameterException",
"message": "Parameter conditions \"_method=ping\" not met for actual request parameters: ",
"path": "/webhooks"
}
Right, I got it working. The answer is a bit tricky so I wanted to register it here should anyone have such problem.
#Neil McGuigan pointed me on the right direction on his comment but I didn't pay attention at first. The main culprit here is a very, very, very bad API design on our remote application's side.
_method is a field used to specify non-standard HTTP verbs such as PUT, PATCH, DELETE, TRACE and so on. This field is filtered by HiddenHttpMethodFilter and the HttpServletRequest is wrapped with this 'new' method. You can see at the file's source how it works.
As I wanted this _method field to get thru the filter without modifying the whole request (and causing the errors because there's no such verb as pingor message on `RequestMethod) I firstly had to deactivate the filter. This could be done by two ways:
I could stop Spring Boot from automagically configuring Spring MVC, skipping WebMvcAutoConfiguration from being loaded when the ApplicationContext was loaded. As you can imagine this is a BIG, BIG, BIIIIG NO because, well, things could happen.
I could use a FilterRegistrationBean to disable the bad filter. Pretty simple and straightforward, this was the method I chose to use:
#Bean
public FilterRegistrationBean registration(HiddenHttpMethodFilter filter) {
FilterRegistrationBean registration = new FilterRegistrationBean(filter);
registration.setEnabled(false);
return registration;
}
Last but not least, I decided to give HiddenHttpMethodFilter a little extension to somehow improve how the requests were getting thru. The Java EE Spec is pretty clear on the Servlet Spec Commandments where it states:
Thou should not alter your request on your side. You must respect the sender (something like that)
Though I agree with this, for the sake of my mental stability I decided to alter it anyway. To achieve this, we can use a simple HttpServletRequestWrapper, override the chosen methods and filter the original request with the wrapped part. I ended up doing something like this:
public class WhatoolsHiddenHttpMethodFilter extends OrderedHiddenHttpMethodFilter {
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
String paramValue = request.getParameter(OrderedHiddenHttpMethodFilter.DEFAULT_METHOD_PARAM);
if("POST".equals(request.getMethod()) && StringUtils.hasLength(paramValue)) {
String method = paramValue.toUpperCase(Locale.ENGLISH);
List<String> whatoolsMethods = Arrays.asList("ping", "message", "carbon", "media", "media_carbon", "ack");
if(whatoolsMethods.contains(paramValue)){
WhatoolsHiddenHttpMethodFilter.HttpMethodRequestWrapper wrapper = new WhatoolsHiddenHttpMethodFilter
.HttpMethodRequestWrapper(request, "POST", paramValue);
filterChain.doFilter(wrapper, response);
} else {
WhatoolsHiddenHttpMethodFilter.HttpMethodRequestWrapper wrapper = new WhatoolsHiddenHttpMethodFilter
.HttpMethodRequestWrapper(request, method, null);
filterChain.doFilter(wrapper, response);
}
} else {
filterChain.doFilter(request, response);
}
}
private static class HttpMethodRequestWrapper extends HttpServletRequestWrapper {
private final String method;
private final String whatoolsMethod;
public HttpMethodRequestWrapper(HttpServletRequest request, String method, String whatoolsMethod) {
super(request);
this.method = method;
this.whatoolsMethod = whatoolsMethod;
}
#Override
public String getMethod() {
return this.method;
}
#Override
public String getHeader(String name) {
if("x-whatools-method".equals(name)){
return this.whatoolsMethod;
}
return super.getHeader(name);
}
#Override
public Enumeration<String> getHeaderNames() {
List<String> names = Collections.list(super.getHeaderNames());
if(this.whatoolsMethod != null){
names.add("x-whatools-method");
}
return Collections.enumeration(names);
}
}
}
So, what this does is to wrap the request with a new x-whatools-method header when the header is in my whatoolsMethods list. With this, I can easily use #RequestMapping's headers property and map the requests to the correct controller methdods.
Back to the initial question, I'm almost sure (well, 99,95% should be completely sure but let's not risk it) the params property on #RequestMapping works only for request parameters on GET URIs, e.g http://foo.bar/?baz=42. It won't work filtering parameters sent on the request's body.
Thanks Neil for your guidance, even if small! I hope this helps someone.
You can use params in a request mapping:
#RequestMapping(value="/foo", params={"_method=ping"})
Assuming these are post parameters that is
params DOES work for POST, I promise you
Here's my controller:
#Controller
#RequestMapping("/test1")
public class ParamTestController {
#RequestMapping(method = RequestMethod.POST)
#ResponseBody String getA(){
return "A";
}
#RequestMapping(method = RequestMethod.POST, params = {"b"})
#ResponseBody String getB(){
return "B";
}
}
Here's my test: