Right now we are holding file in our postgresql database and mapping that content using byte[] field in our entity. I need to investigate if we could
stream the content data direct from the database to the HTTP output stream, and do the same thing in opposite way so stream binary data from HTTP into database using jpa Blob data type. I know that Blob has methods getBinaryStream and setBinaryStream so its may work, and we do not need hold data into memory.
What I am concern are database transaction, because we are mapping entity into DTO, and the second thing is broken Http request and data may be lost in some point.
Is there are anybody who had any experience with that solution ?
Solution for stream-reading data from BLOBs:
Existing BLOB data are streamed by passing OutputStream (provided by the servlet container) into transactional method which writes entity blob data to the stream from inside transaction. Note that the content type of response is set before writing the data.
Entity class:
public class Attachment {
private java.sql.Blob data;
public java.sql.Blob getData() { return data; }
}
Service method:
#Transactional(readOnly = true)
public void copyContentsTo(long attachmentId, OutputStream outputStream) throws IOException {
Attachment dbAttachment = attachmentRepository.findOne(attachmentId);
try (InputStream is = dbAttachment.getData().getBinaryStream()) {
IOUtils.copy(is, outputStream);
} catch (SQLException e) {
throw new ParameterException("Cannot extract BLOB for attachment #" + attachmentId, e);
}
}
REST API Spring Controller method:
#GetMapping(value = "/api/project-attachment/{attachment-id}/content")
#ResponseStatus(HttpStatus.OK)
public void getAttachmentContent(
#PathVariable("attachment-id") long attachmentId,
HttpServletResponse response,
OutputStream stream) throws IOException {
response.setContentType(getMime(attachmentId));
attachmentService.copyContentsTo(attachmentId, stream);
}
Lucasz, Spring Content for JPA does exactly what you are asking. Designed to make it really easy to create Spring applications that handle content (documents, images, video's etc). It supports a range of backend stores one of which being relational DBs and obviously they uses BLOBs.
This JPA module will stream uploaded files from the request input stream directly to the database and vice versa thus it never stores the entire file in-memory which would obviously cause problems with very large files.
It would save you from having to write ANY of the code in #tequilacat's answer.
Probably worth a look.
Related
The spring integration flow I wrote has to get files (some of them are as big as 4G) from a rest service and transfer them to a remote shared drive. For downloading them from the rest service I configured this simple component:
#Bean
public HttpRequestExecutingMessagehandler httpDownloader (RestTemplate template){
Expression expr = (new SpelExpressionParser()).parseExpression("payload.url");
HttpRequestExecutingMessagehandler handler = new HttpRequestExecutingMessagehandler (expr, template);
handler.setExpectedResponseType(byte[].class);
handler.setHttpMethod(GET);
return handler;
}
Unfortunately this won't scale meaning for larger files it will eventually throw java.lang.OutOfMemoryError: Java heap space, even if i add more memory with -Xmx or -XXMaxPermSize
So my question is, what to do in order to avoid these problems no matter how big the files will be?
I think I have answered you in some other similar your question that Spring RestTemplate is not designed for streaming response body. It is explained in this SO thread: Getting InputStream with RestTemplate.
One of the solution which may work for your is to write a custom HttpMessageConverter which would return a File object containing data from HTTP response. This article explains how to do that with the ResponseExtractor, but something like FileHttpMessageConverter is not so hard to implement based on experience from that article. See StreamUtils.copy(InputStream in, OutputStream out)
Then you inject this FileHttpMessageConverter into your HttpRequestExecutingMessagehandler - setMessageConverters(List<HttpMessageConverter<?>> messageConverters).
Your service for remote shared drive should already deal with this local temporary file to get that large content without consuming memory.
See also this one about possible approach via WebFlux: https://www.amitph.com/spring-webclient-large-file-download/
Created this starting from ByteArrayHttpMessageConverter class and injected it into the custom RestTemplate I use. But this solution is based on using a File message, which is not quite the streaming I was hoping for.
public class FileCustomConverter extends AbstractHttpMessageConverter<File> {
public FileCustomConverter() {
super(new MediaType[]{MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL});
}
public boolean supports(Class<?> clazz) {
return File.class == clazz;
}
public File readInternal(Class<? extends File> clazz, HttpInputMessage inputMessage) throws IOException {
File outputFile = File.createTempFile(UUID.randomUUID().toString(), ".tmp");
OutputStream outputStream = new FileOutputStream(outputFile);
StreamUtils.copy(inputMessage.getBody(), outputStream);
outputStream.close();
return outputFile;
}
protected Long getContentLength(File bytes, #Nullable MediaType contentType) {
return bytes.length();
}
protected void writeInternal(File file, HttpOutputMessage outputMessage) throws IOException {
InputStream inputStream = new FileInputStream(file);
StreamUtils.copy(inputStream, outputMessage.getBody());
inputStream.close();
}
}
I have a web socket connection established between a browser client and a Spring Boot backend using STOMP and sock js. Every second, a payload is sent to the server from the client containing data that needs to be persisted to a Postgres database. There may be thousands of clients connected simultaneously so I don't want to update the database every second for each one. So, to reduce the CPU load, I want to listen for when a web socket StompCommand.DISCONNECT event occurs and then persist the last received message from the client.
Is this possible, or is there another way to get around thi sproblem?
In this case - the question is really opinionated - there are many possible implementations.
One of the implementations can do the following:
When you receive the message from the connected client - maintain a map (in memory will be enough, for the sake of idea) of the the identifier of the current client to the Last Data.
Every time you get a new message in #MessageMapping annotated class - update an entry in the map, so that it will always contain the last message.
The value of the map will be the last message, the key can be Principal, SessionId string - whatever you'll find useful.
#Component
public class LastMessageHolder {
private Map<Principal, MyData> lastDataPerPrincipal;
public void updateLastData(Principal principal, MyData data) {
lastDataPerPrincipal.put(principal, data);
}
public MyData getLastDataForPrincipalAndClear(Principal principal) {
return lastDataPerPrincipal.remove(principal);
}
}
The message Receiver will get the messages through the stomp channel and update the last message holder
#Component
public class MyMessageReceiver {
#Autowired
private LastMessageHolder lastMessageHolder;
#MessageMapping(...)
public void onDataReceived(Principal principal, MyData data) {
// this gets called every second per client
lastMessageHolder.updateLastData(principal, data);
}
}
And when you listen for the disconnect message in the channel interceptor - make remove the data from the principal that is being disconnected and store it in the database:
#Component
public class DbStoreChannelInterceptor implements ChannelInterceptor {
#Autowired
private LastMessageHolder lastMessageHolder;
#Autowired // something that will store your stuff in the db
private DbDao dbDao;
#Override
public Message<?> preSend(Message<?> message, MessageChannel channel) {
StompHeaderAccessor accessor =
MessageHeaderAccessor.getAccessor(message,
StompHeaderAccessor.class);
if (StompCommand.CONNECT.equals(accessor.getCommand())) {
// populate a principal here, from headers, authentication token,
whatever
Principal principal = ...
accessor.setUser(principal);
}
if (StompCommand.DISCONNECT.equals(accessor.getCommand())) {
Principal principal = accessor.getUser();
MyData data = lastMessageHolder.getDataForPrincipalAndClear(principal);
dbDao.storeDataInDbForPrincipal(principal, data);
}
}
}
This is a basic idea.
From that you can take it further, and instead of storing the data from the channel interceptor (in this case the actual INSERT will be done for each client) you might want to throw it into some in-memory or distributed queue - whatever suits you best, so that the consumer will read the batch of the data objects and store them all at once, so that it will lead to much less load on your RDBMS.
In addition, I'll just mention, that you should think about the situation where the client keeps sending the data, but the server gets down for some reason, while the client is still interested to keep sending data. This is more in the area of the architecture of the distributed system, so its way beyond the scope of the question.
This question already has answers here:
SpringBoot: Large Streaming File Upload Using Apache Commons FileUpload
(5 answers)
Closed 2 years ago.
Spring boot's default MultiPartResolver interface handles the uploading of multipart files by storing them on the local file system. Before the controller method is entered, the entire multipart file must finish uploading to the server.
We are storing all of our uploaded files directly to a database and our servers have a very small disk quota, so if a large file is uploaded, we are seeing an IOExeption - Disk quota exceeded.
Is there a way to get the stream directly from the client's incoming request before Spring's MultiPartResolver stores the file on the local filesystem so the we can stream directly to our db?
You could use apache directly, as described here https://commons.apache.org/proper/commons-fileupload/streaming.html.
#Controller
public class UploadController {
#RequestMapping("/upload")
public String upload(HttpServletRequest request) throws IOException, FileUploadException {
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iterator = upload.getItemIterator(request);
while (iterator.hasNext()) {
FileItemStream item = iterator.next();
if (!item.isFormField()) {
InputStream inputStream = item.openStream();
//...
}
}
}
}
Make sure to disable springs multipart resolving mechanism.
application.yml:
spring:
http:
multipart:
enabled: false
Actually it is not trivial task. If you would like to write stream from client right to the database, you have to process request manually. There are some libraries, that can make this task simpler. One of them is "Apache Commons FileUpload". Below very simple example, how can you process incoming multipart/form-data request by this library.
#Controller
public class Controller{
#RequestMapping("/upload")
public String upload(HttpServletRequest request){
String boundary = extractBoundary(request);
try {
MultipartStream multipartStream = new MultipartStream(request.getInputStream(),
boundary.getBytes(), 1024, null);
boolean nextPart = multipartStream.skipPreamble();
while(nextPart) {
String header = multipartStream.readHeaders();
if(header.contains("filename")){
//if input is file
OutputStream output = createDbOutputStream();
multipartStream.readBodyData(output);
output.flush();
output.close();
} else {
//if input is not file (text, checkbox etc)
ByteArrayOutputStream output = new ByteArrayOutputStream();
multipartStream.readBodyData(output);
String value = output.toString("utf-8");
//... do something with extracted value
}
nextPart = multipartStream.readBoundary();
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private String extractBoundary(HttpServletRequest request) {
String boundaryHeader = "boundary=";
int i = request.getContentType().indexOf(boundaryHeader)+
boundaryHeader.length();
return request.getContentType().substring(i);
}
}
Header for file field will looks like:
Content-Disposition: form-data; name="fieldName"; filename="fileName.jpg"
Content-Type: image/jpeg
Header for simple field will looks like:
Content-Disposition: form-data; name="fieldName";
Note, that this snippet is just simplified example to show you direction. There is no some details like: extract field name from header, create database output stream etc. You can implement all of this stuff by your own.
Examples of multipart request's field headers you can find in RFC1867. Information about multipart/form-data RFC2388.
I am new to Servlets and Spring Framework.
I try to get media files from directory through Rest Service.
For videos/mp4 I couldn't find anything.
For audio I did this:
Writing mp3 file to response output stream
For images I did this:
#RequestMapping("/tmp/{uuid}")
#ResponseBody
public ResponseEntity<InputStreamResource> getTmp(#PathVariable("uuid") String uuid)
throws IOException {
Path path = Paths.get("/media/psmaster/HDD/TUC-IPS/" + uuid);
String contentType = Files.probeContentType(path);
FileSystemResource file = new FileSystemResource("/media/psmaster/HDD/TUC-IPS/" + uuid);
return ResponseEntity
.ok()
.contentLength(file.contentLength())
.contentType(
MediaType.parseMediaType(contentType))
.body(new InputStreamResource(file.getInputStream()));
}
Can someone please help to figure out the problem?
If you are using Spring 4.2 you can use StreamingResponseBody, Have a look at this post
You can also give Spring Content a look. It allows you to build content services very quickly and easily using similar programming techniques to Spring Data. You can also pair it with Spring Data to additionally store and search metadata for your videos. By defining a single interface and including the appropriate Spring Content dependency in your project you can create a set of REST endpoints that allow you to manage the full lifecycle of a video including streaming.
You can write media using streams and HttpServletResponse:
#RequestMapping(value = "/image/{imgName}", method = RequestMethod.GET)
public void getImageAsByteArray(#PathVariable String imgName , HttpServletResponse response) throws IOException {
InputStream in = servletContext.getResourceAsStream("/WEB-INF/images/" + imgName);
response.setContentType(MediaType.IMAGE_JPEG_VALUE);
IOUtils.copy(in, response.getOutputStream());
}
The example above serves an image file.
Hope this helps
I developed a restful web service via Jersey in Java (JAX-RS) : http://www.vogella.com/articles/REST/article.html
Then I used the Hibernate Technology to map the data to the database.
Finally I developed an android application to display data.
This is an example of a method in my Web Service :
#GET
#Path("/project_id/username/get/{projectId}/{username}/")
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response deliverableList(#PathParam("projectId") long projectId,
#PathParam("username") String username) {
Session session = HibernateUtil.getSessionFactory().getCurrentSession();
session.beginTransaction();
List<Deliverable> list = null;
try {
list= (List<Deliverable>) session.createQuery(
"from Deliverable as d where d.project.id= :id").setLong("id", projectId).list();
} catch (HibernateException e) {
e.printStackTrace();
session.getTransaction().rollback();
}
session.getTransaction().commit();
return Response.status(201).entity(list).build();
}
as you see I used "Response.status(201).entity(list).build()" to transfer the list of data. Is it a good way? if not what is your suggestion to transfer the data. Please support your explanation with some codes and examples.
Response.ok().enity(object).build() is the correct way to return data
You really want to move your hibernate stuff to a data access tier... it's going to be hard to manage intermingled with your service tier
I completely disagree with smcg about using a helper method to map the java to json. Use the jax-rs annotations on your beans to do it unless you have really complicated requirements: see http://wiki.fasterxml.com/JacksonAnnotations
It seems to me like you are relying on something to automagically map your Java objects to JSON - probably Jackson. I do not personally prefer this method. Instead, I use Jettison and create my own mapping from Java to a Jettison JSONObject object. Then I use the JSONObject (or JSONArray) as the entity. My return statement would be something like this:
return Response.ok().entity(myObjectAsJSON).build();
In the case of returning a list of things, use a JSONArray instead of a JSONObject.
You'll want a helper method to map the Java object to JSON.
public JSONArray deliverableListToJSON(List<Deliverable> deliverables)
throws JSONException {
JSONArray result = new JSONArray();
for(Deliverable deliverable : deliverables) {
JSONObject deliverableJSON = new JSONObject();
deliverableJSON.put("importantValue", deliverable.getImportantValue());
result.put(deliverableJSON);
}
return result;
}
This approach gives you more flexibility and doesn't force you to have public getters and setters for all your fields.