I'm currently writing a REST service with spring boot which should provide a file download, i. e. a client application can download files from the service. A file can be several gigabytes big (sometimes bigger than the main memory), i. e. loading the file into the main memory is not an option. So I need some kind of streaming meachanism when sending a file to the client.
One promising solution on the net (taken from here):
#RestController
// ...
#RequestMapping(value = "/files/{file_name}", method = RequestMethod.GET)
#ResponseBody
public FileSystemResource getFile(#PathVariable("file_name") String fileName) {
return new FileSystemResource(myService.getFileFor(fileName));
}
But it does not work. I get:
java.lang.IllegalStateException: Unsupported resource class: class org.springframework.core.io.FileSystemResource
at org.springframework.http.converter.ResourceHttpMessageConverter.readInternal(ResourceHttpMessageConverter.java:100)
I researched but don't know what is causing the issue. I tried other approaches (e. g. Downloading a file from spring controllers) but I get the same error. Does anyone know why the solutions seem to work for others but not for me?
Edit: Here is where I call the REST service (jUnit test case):
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
public class DownloadTest {
#LocalServerPort
private int port;
private RestTemplate template;
private String requestPath;
#Before
public void setUp() throws Exception {
template = new RestTemplate();
requestPath = "http://localhost:" + port + "/files/download";
}
#Test
public void test() {
Resource FileSystemResource =
template.getForObject(requestPath, FileSystemResource.class);
}
}
The code of the RestController is working as expected with a small memory footprint (the file is never loaded to the main memory completely).
The client code that works for me (also with a small memory footprint) can be found here.
i fixed the java.lang.IllegalStateException: Unsupported resource error changing this
Resource FileSystemResource =
template.getForObject(requestPath, FileSystemResource.class);
for this
Resource FileSystemResource =
template.getForObject(requestPath, Resource .class);
Related
Service class :
#Value("classpath:multipleHearingLocations.json")
public static Resource resource;
public void someMethod(){
ConcurrentHashMap<String,List<String>> multipleHearingLocations = new Objec`enter code here`tMapper().readValue(Files.newInputStream(Paths.get(**resource.getURI()**)), new TypeReference<ConcurrentHashMap<String, List<String>>>(){});
}
someTest.json - resided in my main/resources folder
While testing in Jusint5, test class
#Value("classpath:multipleHearingLocations.json")
public static Resource resource;
But when I test it - the service calss throws a null pointer exception saying resource is null.
Test Method :
#Mock
public static Resource resource;
enter code here
#Test
void getMultipleHearingLocations_shouldReturnCorrespondingMultipleEpimsIdForVenue() throws IOException {
SscsCaseData caseData = SscsCaseData.builder()
.appeal(Appeal.builder()
.hearingOptions(HearingOptions.builder().build())
.build())
.processingVenue(PROCESSING_VENUE_1)
.build();
// = new ClassPathResource("multipleHearingLocationsTest.json");
given(venueService.getEpimsIdForVenue(caseData.getProcessingVenue())).willReturn(Optional.of("443014"));
given(referenceDataServiceHolder.getVenueService()).willReturn(venueService);
given(**resource.getURI()**).willReturn(new ClassPathResource("multipleHearingLocationsTest.json").getURI());
List<HearingLocation> result = HearingsDetailsMapping.getHearingLocations(
caseData,
referenceDataServiceHolder
);
What is it that I am doing wrong?
Or is there a way to inject the Resource in the tests as well, so that it is available as a bean when the service class is called?
Tried running it as below, still no luck
RunWith(SpringRunner.class)
#SpringBootTest
#ContextConfiguration(classes = HearingsDetailsMapping.class)
#TestPropertySource(locations = "classpath:multipleHearingLocationsTest.json")
Any help will be appreciated :)
I've encounter the same problem. This injection method works fine with JUnit 4 but not with Junit5.
#RunsWith is a JUnit 4 annotation the JUnit 5 equivalent is #ExtendWith(SpringExtension.class) see this answer for more details.
To inject a resource in JUnit 5 you could use this working soulution
or
File getFile(String fileLocation) {
return new File(this
.getClass()
.getClassLoader()
.getResource(fileLocation)
.getFile());
}
String filelocation = "multipleHearingLocations.json" //(no "class:" before path)
Path to multipleHearingLocations.json while testing: src/test/resources/multipleHearingLocations.json
I want to upload a multipart file to AWS S3. So, i have to convert it.
But new File method needs a local location to get the file.
I am able to do in local. But running this code in every machine seems like a issue.
Please find both scenarios.
Working
private File convertMultiPartToFile(MultipartFile multipartFile) throws IOException {
File convFile = new File("C:\\Users\\" + multipartFile.getOriginalFilename());
multipartFile.transferTo(convFile);
return convFile;
}
Not working
private File convertMultiPartToFile(MultipartFile multipartFile) throws IOException {
File convFile = new File(multipartFile.getOriginalFilename());
multipartFile.transferTo(convFile);
return convFile;
}
Error received :
java.io.FileNotFoundException: newbusiness.jpg (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
You could use Spring Content S3. This will hide the implementation details so you don't need to worry about them.
There are Spring Boot starter alternatives but as you are not using Spring Boot add the following dependency to your pom.xml
pom.xml
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-s3</artifactId>
<version>0.0.11</version>
</dependency>
Add the following configuration that creates a SimpleStorageResourceLoader bean:
#Configuration
#EnableS3Stores
public class S3Config {
#Autowired
private Environment env;
public Region region() {
return Region.getRegion(Regions.fromName(env.getProperty("AWS_REGION")));
}
#Bean
public BasicAWSCredentials basicAWSCredentials() {
return new BasicAWSCredentials(env.getProperty("AWS_ACCESS_KEY_ID"), env.getProperty("AWS_SECRET_KEY"));
}
#Bean
public AmazonS3 client(AWSCredentials awsCredentials) {
AmazonS3Client amazonS3Client = new AmazonS3Client(awsCredentials);
amazonS3Client.setRegion(region());
return amazonS3Client;
}
#Bean
public SimpleStorageResourceLoader simpleStorageResourceLoader(AmazonS3 client) {
return new SimpleStorageResourceLoader(client);
}
}
Create a "Store":
S3Store.java
public interface S3Store extends Store<String> {
}
Autowire this store into where you need to upload resources:
#Autowired
private S3Store store;
WritableResource r = (WritableResource)store.getResource(getId());
InputStream is = // plug your input stream in here
OutputStream os = r.getOutputStream();
IOUtils.copy(is, os);
is.close();
os.close();
When your application starts it will see the dependency on spring-content-s3 and your S3Store interface and inject an implementation for you, therefore, you don't need to worry about implementing this yourself.
IF you writing some sort of web application or microservice and you need a REST API then you can also add this dependency:
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-rest</artifactId>
<version>0.0.11</version>
</dependency>
Update your S3Config.java as follows:
#Configuration
#EnableS3Stores
#Import(RestConfiguration.class)
public class S3Config {
...
Update your store as follows:
S3Store.java
#StoreRestResource(path="s3docs")
public interface S3Store extends Store<String> {
}
Now when your application starts it will see your Store interface and also inject an #Controller implementation that will forward REST request onto your store. This replaces the autowiring code above obviously.
Then:
curl -X POST /s3docs/example-doc
with a multipart/form-data request will store the image in s3.
curl /s3docs/example-doc
will fetch it again and so on. This controller supports full CRUD and video streaming by the way.
If you want to associate this "content" with JPA Entity or something like that then you can have your S3Store extend AssociateStore or ContentStore and you have additional methods available that provide for associations.
There are a couple of getting started guides here. The s3 reference guide is here. And there is a tutorial video here. The coding bit starts about 1/2 way through.
HTH
Since it needs a temporary location to place files. Below code worked after deploying war on AWS.
private File convertMultiPartToFile(MultipartFile multipartFile) throws IOException {
File convFile = new File(System.getProperty("java.io.tmpdir") + System.getProperty("file.separator") +
multipartFile.getOriginalFilename());
multipartFile.transferTo(convFile);
return convFile;
}
You have problems with relative Paths
You can do this
public class UploadStackoverflow {
private String location = "upload-dir";
private Path rootLocation;
public File convertFile(MultipartFile file) throws IOException {
rootLocation = Paths.get(location);
Files.createDirectories(rootLocation);
String filename = StringUtils.cleanPath(file.getOriginalFilename());
InputStream inputStream = file.getInputStream();
Files.copy(inputStream, this.rootLocation.resolve(filename),
StandardCopyOption.REPLACE_EXISTING);
return new File(this.rootLocation.resolve(filename).toAbsolutePath().toString());
}
}
Using apachebench with "ab -k -c 50 -n 1000000" options (50 concurrent threads) shows a 10x performance difference between the following 2 methods (manual and spring-managed serialization). Is it possible to achieve the same performance via configuration of Spring serialization?
I'm running the test on Windows 7, JDK8, i7-6700. Embedded Tomcat, similar results with Undertow or Jetty too. A similar WildFly 10 JAX-RS sample apps performance yields similar results as the manual spring one, so I see no reason why Spring automatic mode should be so slow.
Full source code:
#SpringBootApplication
#Controller
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
ObjectMapper mapper = new ObjectMapper(new JsonFactory());
#RequestMapping(value = "/auto", produces = "application/json; charset=utf-8")
#ResponseBody
public Lol automaticSerialization() {
Lol lol = new Lol();
lol.a = UUID.randomUUID().toString();
lol.b = System.currentTimeMillis();
return lol;
}
#RequestMapping(value = "/manual", produces = "application/json; charset=utf-8")
#ResponseBody
public String manualSerialization() throws JsonProcessingException {
Lol lol = new Lol();
lol.a = UUID.randomUUID().toString();
lol.b = System.currentTimeMillis();
return mapper.writeValueAsString(lol);
}
public static class Lol {
String a;
long b;
public void setA(String a) {
this.a = a;
}
public void setB(long b) {
this.b = b;
}
public String getA() {
return a;
}
public long getB() {
return b;
}
}
}
Edit:
Trace of automatic serialization:
Trace of manual serialization:
The only idea that I have is that Spring's default ObjectMapper is configured a bit differently than the one you use in your benchmark. Like the comments mention, you'd probably see a bit of overhead if you let Spring handle the mapping automatically but it shouldn't have more than a few percent's worth of impact.
To be sure that the comparison is fair, add this bean definition to your configuration:
#Bean
#Primary
ObjectMapper objectMapper() {
return new ObjectMapper(new JsonFactory());
}
and replace ObjectMapper mapper = new ObjectMapper(new JsonFactory()); with an autowired field:
#Autowired
ObjectMapper mapper;
and see if the benchmarks return the same value.
EDIT
I wanted to verify this for myselt so I wrote a JMeter plan and executed each endpoint exactly 5kk times, with a 1-minute warm-up period. The results were as expected, no major differences between the approaches:
Label,# Samples,Average,Min,Max,Std. Dev.,Error %,Throughput,KB/sec,Avg. Bytes
Auto Request,5000000,2,0,108,5.88,0.00%,15577.3,3088.08,203.0
Manual Request,5000000,2,0,149,5.99,0.00%,15660.2,2813.94,184.0
The important thing to note is the throughput difference - auto's 15577.3 vs. manual's 15660.2.
Here's my JMeter test plan, if you'd like to test it yourself, I was running on port 8081. If I find the time, I'll try another benchmarking framework, perhaps Gatling.
I am experiencing problems when configurating my Jersey Client with the ApacheConnector. It seems to ignore all request headers that I define in a WriterInterceptor. I can tell that the WriterInterceptor is called when I set a break point within WriterInterceptor#aroundWriteTo(WriterInterceptorContext). Contrary to that, I can observe that the modification of an InputStream is preserved.
Here is a runnable example demonstrating my problem:
public class ApacheConnectorProblemDemonstration extends JerseyTest {
private static final Logger LOGGER = Logger.getLogger(JerseyTest.class.getName());
private static final String QUESTION = "baz", ANSWER = "qux";
private static final String REQUEST_HEADER_NAME_CLIENT = "foo-cl", REQUEST_HEADER_VALUE_CLIENT = "bar-cl";
private static final String REQUEST_HEADER_NAME_INTERCEPTOR = "foo-ic", REQUEST_HEADER_VALUE_INTERCEPTOR = "bar-ic";
private static final int MAX_CONNECTIONS = 100;
private static final String PATH = "/";
#Path(PATH)
public static class TestResource {
#POST
public String handle(InputStream questionStream,
#HeaderParam(REQUEST_HEADER_NAME_CLIENT) String client,
#HeaderParam(REQUEST_HEADER_NAME_INTERCEPTOR) String interceptor)
throws IOException {
assertEquals(REQUEST_HEADER_VALUE_CLIENT, client);
// Here, the header that was set in the client's writer interceptor is lost.
assertEquals(REQUEST_HEADER_VALUE_INTERCEPTOR, interceptor);
// However, the input stream got gzipped so the WriterInterceptor has been partly applied.
assertEquals(QUESTION, new Scanner(new GZIPInputStream(questionStream)).nextLine());
return ANSWER;
}
}
#Provider
#Priority(Priorities.ENTITY_CODER)
public static class ClientInterceptor implements WriterInterceptor {
#Override
public void aroundWriteTo(WriterInterceptorContext context)
throws IOException, WebApplicationException {
context.getHeaders().add(REQUEST_HEADER_NAME_INTERCEPTOR, REQUEST_HEADER_VALUE_INTERCEPTOR);
context.setOutputStream(new GZIPOutputStream(context.getOutputStream()));
context.proceed();
}
}
#Override
protected Application configure() {
enable(TestProperties.LOG_TRAFFIC);
enable(TestProperties.DUMP_ENTITY);
return new ResourceConfig(TestResource.class);
}
#Override
protected Client getClient(TestContainer tc, ApplicationHandler applicationHandler) {
ClientConfig clientConfig = tc.getClientConfig() == null ? new ClientConfig() : tc.getClientConfig();
clientConfig.property(ApacheClientProperties.CONNECTION_MANAGER, makeConnectionManager(MAX_CONNECTIONS));
clientConfig.register(ClientInterceptor.class);
// If I do not use the Apache connector, I avoid this problem.
clientConfig.connector(new ApacheConnector(clientConfig));
if (isEnabled(TestProperties.LOG_TRAFFIC)) {
clientConfig.register(new LoggingFilter(LOGGER, isEnabled(TestProperties.DUMP_ENTITY)));
}
configureClient(clientConfig);
return ClientBuilder.newClient(clientConfig);
}
private static ClientConnectionManager makeConnectionManager(int maxConnections) {
PoolingClientConnectionManager connectionManager = new PoolingClientConnectionManager();
connectionManager.setMaxTotal(maxConnections);
connectionManager.setDefaultMaxPerRoute(maxConnections);
return connectionManager;
}
#Test
public void testInterceptors() throws Exception {
Response response = target(PATH)
.request()
.header(REQUEST_HEADER_NAME_CLIENT, REQUEST_HEADER_VALUE_CLIENT)
.post(Entity.text(QUESTION));
assertEquals(200, response.getStatus());
assertEquals(ANSWER, response.readEntity(String.class));
}
}
I want to use the ApacheConnector in order to optimize for concurrent requests via the PoolingClientConnectionManager. Did I mess up the configuration?
PS: The exact same problem occurs when using the GrizzlyConnector.
After further research, I assume that this is rather a misbehavior in the default Connector that uses a HttpURLConnection. As I explained in this other self-answered question of mine, the documentation states:
Whereas filters are primarily intended to manipulate request and
response parameters like HTTP headers, URIs and/or HTTP methods,
interceptors are intended to manipulate entities, via manipulating
entity input/output streams
A WriterInterceptor is not supposed to manipulate the header values while a {Client,Server}RequestFilter is not supposed to manipulate the entity stream. If you need to use both, both components should be bundled within a javax.ws.rs.core.Feature or within the same class that implements two interfaces. (This can be problematic if you need to set two different Prioritys though.)
All this is very unfortunate though, since JerseyTest uses the Connector that uses a HttpURLConnection such that all my unit tests succeeded while the real life application misbehaved since it was configured with an ApacheConnector. Also, rather than suppressing changes, I wished, Jersey would throw me some exceptions. (This is a general issue I have with Jersey. When I for example used a too new version of the ClientConnectionManager where the interface was renamed to HttpClientConnectionManager I simply was informed in a one line log statement that all my configuration efforts were ignored. I did not discover this log statement til very late in development.)
I followed the gwt 2.4 validation sample and implemented the whole stuff into my own App. The client side works great.
private void verifyRegistrationData(final RegistrationTO registration) throws ConstraintViolationException {
final Validator validator = Validation.buildDefaultValidatorFactory().getValidator();
final Set<ConstraintViolation<RegistrationTO>> violations = validator.validate(registration);
if (violations.size() > 0) {
final Set<ConstraintViolation<?>> temp = new HashSet<ConstraintViolation<?>>(violations);
throw new ConstraintViolationException(temp);
...
but if I do the same on the server side:
public void update(final RegistrationTO registration) throws IllegalArgumentException, ConstraintViolationException, TestException {
final Set<ConstraintViolation<RegistrationTO>> violations = validator.validate(registration);
if (!violations.isEmpty()) {
final Set<ConstraintViolation<?>> temp = new HashSet<ConstraintViolation<?>>(violations);
throw new ConstraintViolationException(temp);
}
...
the whole thing crashes with the following exception:
javax.servlet.ServletContext log: Exception while dispatching incoming RPC call
com.google.gwt.user.client.rpc.SerializationException: Type 'org.hibernate.validator.engine.PathImpl' was not included in the set of types which can be serialized by this SerializationPolicy or its Class object could not be loaded. For security purposes, this type will not be serialized.
That's how PathImpl looks like hibernate-validator-4.1.0.Final-sources.jar
public class PathImpl implements Path, Serializable {
private static final long serialVersionUID = 7564511574909882392L;
...
looks OK (at least to me)
I am using GWT 2.4, validation-api-1.0.0.GA, hibernate-validator-4.1.0.Final, gwt-servlet-deps ...
Thanks in advance!
Is there an explicitly defined a default constructor? i.e.,
public PathImpl() { } ? This is required by GWT's serialization mechanism; if it isn't in the source, serializing an RPC response will fail.
A custom serializer does exist for PathImpl, it's just that unless that class is explicitly referenced in your service API, it's not going to be added to the serialization policy.
The current work around is to add a dummy PathImpl field somewhere in your service API. The ValidationSupport class exists to group this and other such classes together to make this a bit easier.
I change the whole thing to RequestFactory as Thomas Broyer recommended. It was by far not so easy as GWT-RPC. This was the reason for me to collect all kind of informations and to build a sample program. For those who are interested - here you can find a sample with documentation and source. (Single line client logger is also implemented) (Documentation is in German but logging-output aso. is in English...)