I'm running on the IBM public cloud. I have apu connect to access the cloud foundry microservice. I've gone through many of the posts and tried various things and I can't seem to get this to work. Here are my property file config settings for spring boot:
# The name of the application
spring.application.name=xxxxx
# web base path
management.endpoints.web.base-path=/
# Embedded tomcat config
server.tomcat.max-swallow-size=256MB
server.tomcat.max-http-post-size=256MB
# File size values
spring.servlet.multipart.max-file-size=256MB
spring.servlet.multipart.max-request-size=256MB
spring.servlet.multipart.enabled=true
# Server specific values
input.server=xxx
input.rtm.bucket=xxx
storage.server.base=xxx
# Cloudant database info
input.events.db.name=xxxx
input.ait.info.db.name=xxxx
letter.number.db.name=xxxx
letter.gen.data.db.name=xxxx
# Query index design documents
query.pad.ait.info.index.name=xxxx
query.pad.ait.info.deisgn.doc=_xxxx
query.rfa.ltr.index.name=xxxx
query.rfa.ltr.design.doc=xxxx
# The logging levels of the application
logging.level.application=DEBUG
#logging.level.root=DEBUG
#logging.level.org.springframework.web=INFO
# Testing
unit.testing=false
integration.testing=true
# Jackson json config
spring.jackson.mapper.accept-case-insensitive-properties=true
Here is the REST api function for POSTing the file
#PostMapping(value = "/send/rtm/document/{errata}")
public #ResponseBody ResponseEntity<Object> receiveRtmDocument(#PathVariable("errata") String errata, #RequestParam("file") MultipartFile file)
I'm using spring boot 2.1.6 and have not updated anything in the POM file. I'm attempting to send a 5.8 MB file to the api and it gives me this error:
com.ibm.tools.cloud.exceptions.DataNotJsonException: <html>
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor="white">
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>openresty</center>
</body>
</html>
at com.ibm.msc.gasm.sapt.input.AitInputManagement.sendRtmDocument(AitInputManagement.java:182)
at com.ibm.msc.gasm.sapt.test.InputServiceTester.performTest(InputServiceTester.java:142)
at com.ibm.msc.gasm.sapt.test.InputServiceTester.main(InputServiceTester.java:96)
Here is the send code I am using in java for the multipart. The only other headers I use that are not listed here are my authorization headers.
// Create the URL connection
HttpURLConnection conn = (HttpURLConnection) (new URL(requestUri)).openConnection();
if (content != null || multipartFile) conn.setDoOutput(true);
conn.setRequestMethod(method.toString());
// Set the headers
Enumeration<String> keys = headers.keys();
while (keys.hasMoreElements())
{
// Pull out the key
String key = keys.nextElement();
// Set the header
conn.setRequestProperty(key, headers.get(key));
}
// Set the accept header
if (acceptHeader != null) conn.setRequestProperty("Accept", acceptHeader);
// Set the content header
if (contentTypeHeader != null) conn.setRequestProperty("Content-Type", contentTypeHeader);
if (content != null)
{
// Set the content
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
if (content.isFileContent()) dos.write(content.getFileContentAsByteArray());
else if (content.isByteArrayContent()) dos.write(content.getContentAsByteArray());
else if (content.isStringContent()) dos.write(content.getStringContentAsByteArray());
// close the stream
dos.flush();
dos.close();
}
// Set the multipart file
if (multipartFile)
{
// Set the properties
conn.setUseCaches(false);
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Cache-Control", "no-cache");
conn.setRequestProperty("Content-Type", "multipart/form-data;boundry=" + MP_BOUNDRY);
// Set the content
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
dos.writeBytes(MP_HYPHENS + MP_BOUNDRY + StringUtils.crlf);
dos.writeBytes("Content-Disposition: form-data: name=\"" + this.mpName + "\";filename=\"" + this.mpFileName + "\"" + StringUtils.crlf);
dos.writeBytes(StringUtils.crlf);
dos.write(IOUtils.toByteArray(new FileInputStream(this.mpFileNamePath)));
dos.writeBytes(StringUtils.crlf);
dos.writeBytes(MP_HYPHENS + MP_BOUNDRY + MP_HYPHENS + StringUtils.crlf);
// close the stream
dos.flush();
dos.close();
}
// Get the response
HttpResponseMessage response = null;
try
{
// Extract the stream
InputStream is = (conn.getResponseCode() >= HttpURLConnection.HTTP_BAD_REQUEST) ? conn.getErrorStream() : conn.getInputStream();
// Pull out the information
byte[] data = IOUtils.toByteArray(is);
// Set the response
response = new HttpResponseMessage(requestUri, HttpStatusCode.getType(conn.getResponseCode()), acceptHeader, data, conn.getResponseMessage());
}
catch (Throwable e)
{
throw new IOException(String.format("Error reading results from %s", requestUri), e);
}
// Close the request
conn.disconnect();
// Send request
return response;
I've tried several things, but I am not sure what I am missing. Anyone have any ideas how to fix this?
You need to change NGINX settings;
Add to config file next line
client_max_body_size 20M;
Use the form form to submit the file and accept it with MultipartFile. In this case (the other situation is not clear), the default file size is limited to 2M. If you want to upload a large file, you need to configure the file size.
https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/
Try these two in your application.properties
server.tomcat.max-swallow-size=XMB //maximum size of the request body/payload
server.tomcat.max-http-post-size=XMB //maximum size of entire POST request
X is your desired integer representing megabyte.
Related
I am developing a proxy service to a Minio server using WebClient that handles all Minio/S3 API endpoints. Most of them work fine, but I have encountered one case in which the PUT operation seems to get hung up when trying to set the body of the request to either an InputStream, a File, or a Resource pointing to it. (See epilogue at the bottom, as I'm left wondering where the problem really is.)
The only way I've found to make it work is to read the file contents to an in-memory byte array. The following baseline works, for example:
WebClient.UriSpec<WebClient.RequestBodySpec> uriSpec = client.method(request.getMethod());
WebClient.RequestBodySpec bodySpec = uriSpec.uri(uri);
WebClient.RequestHeadersSpec<?> headersSpec = bodySpec;
try {
// read file to byte array; works fine
byte[] bytes = Files.readAllBytes(Path.of(file.get().getFile().toURI()));
// set it to the request body
headersSpec = bodySpec.bodyValue(bytes);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
// manipulate some headers
headersSpec = headersSpec.headers(httpHeaders -> ...);
// execute the request; works fine in this scenario
return headersSpec.exchangeToMono(resp -> ...)
.doOnError(throwable -> log.error("Trouble proxying request: " + throwable.getMessage(), throwable));
However, every alternative that I try to stream this content instead, results in a request that seems to hang in the headersSpec.exchangeToMono invocation. I don't see any errors on the proxy service, and the client socket eventually gives up:
java.net.SocketTimeoutException: timeout
client-tester_1 | at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:143) ~[okio-jvm-2.8.0.jar:na]
client-tester_1 | Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Some examples of failure (or, paraphrasing Edison, I've successfully found at least a dozen ways that do not work):
// Use same byte array as above; Hangs
Resource resource = new ByteArrayResource(bytes);
headersSpec = bodySpec.bodyValue(resource);
// Read an input stream from the file (this one relies on a HttpMessageWriter<InputStream> that I configured on the client); Hangs
InputStream bodyStream = new BufferedInputStream(Files.newInputStream(Path.of(file.get().getFile().toURI())));
headersSpec = bodySpec.bodyValue(bodyStream);
// Resource for the file; Hangs
Resource resource = new FileSystemResource(Path.of(file.get().getFile().toURI()));
Flux<DataBuffer> flux = DataBufferUtils.read(resource, DefaultDataBufferFactory.sharedInstance, 4096);
headersSpec = bodySpec.body(flux, DataBuffer.class);
// Different resource; Hangs
Resource resource = new UrlResource(file.get().getFile().toURI());
headersSpec = bodySpec.bodyValue(resource);
// Try BodyInserters; Hangs
Flux<DataBuffer> flux = DataBufferUtils.read(Path.of(file.get().getFile().toURI()), DefaultDataBufferFactory.sharedInstance, 4096);
headersSpec = bodySpec.body(BodyInserters.fromDataBuffers(flux));
// Yet another attempt; Take a guess...
InputStream bodyStream = new BufferedInputStream(Files.newInputStream(Path.of(file.get().getFile().toURI())));
headersSpec = bodySpec.body(BodyInserters.fromResource(resource));
I'm using recent versions of the relevant libraries:
org.springframework.boot:spring-boot-starter-webflux -> 2.7.5
org.springframework.boot:spring-boot-starter-reactor-netty:2.7.5
org.springframework:spring-core:5.3.23
Epiloge I'm wondering if the problem is not necessarily with Spring/WebClient/Netty -- as many of these code samples were inspired by other examples I've found -- but rather by some nuance on the Minio server?
I have a .net core 2.1 api application that will download a file from a remote location based on the file name. Here is the code:
static public class FileDownloadAsync
{
static public async Task DownloadFile(string filename)
{
//File name is 1GB.zip for testing
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
using (HttpClient client = new HttpClient())
{
string url = #"http://speedtest.tele2.net/" + filename;
using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
using (Stream readFrom = await response.Content.ReadAsStreamAsync())
{
string tempFile = $"D:\\Test\\{filename}";
using (Stream writeTo = File.Open(tempFile, FileMode.Create))
{
await readFrom.CopyToAsync(writeTo);
}
}
stopwatch.Stop();
Debug.Print(stopwatch.Elapsed.ToString());
}
}
}
This is working great, it will pull a 1 gig file down in about 50 seconds. Well within the required download time. I have hard coded a test file to download in this code for testing as well as storage location--these values will ultimately come from a config file when moved into production. Here is the API endpoint that calls this function:
[HttpGet("{fileName}")]
public async Task<string> GetFile(string fileName)
{
await FileDownloadAsync.DownloadFile(fileName);
return "Done";
}
So getting the file from a remote location down to the local server is not a problem. I need some help/guidance on re-posting this file to another API. Once the file is downloaded, there is some work done on the file to prepare it for upload (the files are all MP4 files), and once that work is done, I need to post it to another API for more proprietary processing. Here is the API end point data I have:
POST: /batch/requests Allocates resources to start new batch transcription. Use this method to request[work] on the input
audio data. Upon the accepted request, the response provides
information about the associated request ID and processing status.
Headers: Authorization: Authorization token
Accept: application/json
Content-Type: Indicates the audio format. The value must be:
audio/x-wav;codec=pcm;bit=16;rate=8000;channels=1
audio/x-wav;codec=pcm;bit=16;rate=16000;channels=1
audio/x-raw;codec=pcm;bit=16;rate=8000;channels=1
audio/x-raw;codec=pcm;bit=16;rate=16000;channels=1
video/mp4
Content-Length (optional): The size of the input voice file. Not
required if a chunked transfer is used.
Query string parameters (required):
profileId: one of supported (see GET profiles) customerId: the id of
the customer. A string of minimum 1 and up to 250 alphanumeric, dot
(.) and dash (-) characters.
So I will set the Content-Type to video/MP4 for processing. Note that if the input size is not used if a chunked transfer is used.
Right now, I am more concerned with just posting (streaming) the file in a non-chunked format while we await for more information on what they consider "chunking" a file.
So I am looking for help on steaming the file from disk to the endpoint. Everything I am running across for .net core API is creating the API to download the file from a POST like a Razor page or Angular page--I already have that. I just need some help on "re-posting" to another API.
Thanks
Using the HttpClient you open a stream to the file, create a content stream, set the necessary headers and post to the endpoint
Stream file = File.Open(filepath, FileMode.Open);
var content = new StreamContent(file);
content.Headers.ContentType = new MediaTypeHeaderValue("video/MP4");
client.DefaultRequestHeaders.Add("Authorization", "token here");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json");
using (HttpResponseMessage response = await client.PostAsync(url, content)) {
//...
}
The project I'm working on needs to support large file uploads and know the time taken during their upload.
To handle the large files I'm using the streaming API of Apache FileUpload, this also allows me to measure the time taken for the complete stream to be saved.
The problem I'm having is that I cannot seem to be able to utilise MockMvc in an Integration Test on this controller. I know that the controller works as I've successfully uploaded files using postman.
Simplified Controller Code:
#PostMapping("/upload")
public String handleUpload(HttpServletRequest request) throws Exception {
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iterStream = upload.getItemIterator(request);
while (iterStream.hasNext()) {
FileItemStream item = iterStream.next();
String name = item.getFieldName();
InputStream stream = item.openStream();
if (!item.isFormField()) {
// Process the InputStream
} else {
String formFieldValue = Streams.asString(stream);
}
}
}
Simplified Test Code:
private fun uploadFile(tfr: TestFileContainer) {
val mockFile = MockMultipartFile("file", tfr.getData()) // .getData*() returns a ByteArray
val receiveFileRequest = MockMvcRequestBuilders.multipart("/upload")
.file(mockFile)
.contentType(MediaType.MULTIPART_FORM_DATA)
val result = mockMvc.perform(receiveFileRequest)
.andExpect(status().isCreated)
.andExpect(header().exists(LOCATION))
.andReturn(
}
This is the error I'm currently getting
org.apache.tomcat.util.http.fileupload.FileUploadException: the
request was rejected because no multipart boundary was found
Can anyone help?
The MockMultipartFile approach won't work as Spring does work behind the scenes and simply passes the file around.
Ended up using RestTemplate instead as it actually constructs requests.
I have a Springboot REST application that downloads files from a given directory.
The downloads can be any file file and have any format, and I want to use the original filename as the filename of the downloaded file.
I used the code below to set the filename in the header, and add the header to the response:
#RestController
#RequestMapping("/downloads")
public class DownloadCsontroller {
...
#GetMapping
public void downloadSingleFile(#RequestParam("file") String filename, HttpServletResponse response) throws IOException {
String filepath = m_attachmentPathLocation + File.separator + filename;
File file = new File(filepath);
String contentType = getContentType(file);
response.setStatus(HttpServletResponse.SC_OK);
response.setContentType(contentType);
response.setHeader("Content-Disposition:", "attachment;filename=\"" + file.getName() + "\"");
...
}
...
}
Tested using both "Content-Disposition" and "Content-Disposition:" in setHeader().
Almost everything works (file types), except for PDF, ZIP, RAR, EXE, etc.
Any files (types) not on the list can be downloaded with the desired filenames.
But when any of the file download (PDF, ZIP, RAR, EXE, etc)... it seems it continuously loads like forever... and I cannot even see any request sent in POSTMAN, inspector, firebug, etc.
If I comment out:
//response.setHeader("Content-Disposition:", "attachment;filename=\"" + file.getName() + "\"");
It would work, but the filename would be set to the name of the request mapping. which in this case is "downloads".
I have seen lots of samples that uses "Content-Disposition" header to change the attachment filename... but it seems it fails on these file types.
I have no configurations yet, and it is kinda weird since in most samples I searched... this should be running or working.
TIA
Please add #GetMapping(produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
and instead of returning direct file try to return stream.
Also make a note that "Content-Disposition:" header will not work if the requesting app IP & Port number is different from server app IP & Port number.
Things would work If you can alter the code a bit, by setting all header values using org.springframework.http.HttpHeaders class.
Now looking at your code, i suspect you trying to expose an API that allows to download a multipart File.
I would suggest you not to use HttpServletResponse Class to set the Content- Dispositionheader but use HttpHeaders class. Below is the reformatted code
#RestController
public class DownloadCsontroller {
#GetMapping(value="/downloads")
public ResponseEntity<Object> downloadSingleFile(#RequestParam("file")
String filename) throws IOException {
String filepath = m_attachmentPathLocation + File.separator + filename;
File file = new File(filepath);
String contentType = getContentType(file);
/* response.setStatus(HttpServletResponse.SC_OK);
response.setContentType(contentType);
response.setHeader("Content-Disposition:", "attachment;filename=\""
+ file.getName() + "\"");
*/
// Here is the below Code you need to reform for Content-
//Disposition and the remaining header values too.
HttpHeaders headers= new HttpHeaders();
headers.add("Content-Disposition", "attachment; filename
=whatever.pdf");
headers.add("Content-Type",contentType);
// you shall add the body too in the ResponseEntity Return object
return new ResponseEntity<Object>(headers, HttpStatus.OK);
}
}
Having built an app using PCL method in Xamarin and have had it working 100% using standard HTTP I now changed the remote test server to use SSL with self signed certs.
The app contacts a custom API for logging onto a server and querying for specific data.
I've changed the app to look at SSL now and initially got an error regarding Authentication not working or something but turned off SSL related errors for testing using:
ServicePointManager.ServerCertificateValidationCallback += (o, certificate, chain, errors) => true;
in my AppDelegate files FinishedLaunching method which got over that error.
I'm now getting a 404 / protocol error when trying to do my Login POST to the given URL.
I am using HttpWebRequest for my RESTful calls and this works fine if I change back to plain http.
Not sure why but some articles suggested using ModernHttpClient, which I did. I imported the component (also added the package using NuGet) to no avail.
Am I missing something else that I should be configuring in my code related to httpwebresponse when contacting the SSL server or is this component simply incapable of speaking to an SSL server?
My login function is as follows (Unrelated code removed/obfuscated):
public JsonUser postLogin(string csrfToken, string partnerId, string username, string password){
string userEndPoint = SingletonAppSettngs.Instance ().apiEndPoint;
userEndPoint = userEndPoint.Replace ("druid/", "");
var request = WebRequest.CreateHttp(string.Format(this.apiBaseUrl + userEndPoint + #"user/login.json"));
// Request header collection set up
request.ContentType = "application/json";
request.Headers.Add ("X-CSRF-Token", csrfToken);
// Add other configs
request.Method = "POST";
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
string json_body_content = "{\"username\":\"" + username + "\",\"password\":\"" + password + "\"}";
streamWriter.Write(json_body_content);
streamWriter.Flush();
streamWriter.Close();
}
try{
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (StreamReader reader = new StreamReader (httpResponse.GetResponseStream ())) {
var content = reader.ReadToEnd ();
content = content.Replace ("[],", "null,");
content = content.Replace ("[]", "null");
if (content == null) {
throw new Exception ("request_post_login - content is NULL");
} else {
JsonSerializerSettings jss = new JsonSerializerSettings();
jss.NullValueHandling = NullValueHandling.Ignore;
JsonUser deserializedUser = JsonConvert.DeserializeObject<JsonUser>(content, jss);
if(content.Contains ("Hire company admin user")){
deserializedUser.user.roles.__invalid_name__5 = "Hire company admin user";
deserializedUser.user.roles.__invalid_name__2 = "authenticated user";
}
return deserializedUser;
}
}
}catch(Exception httpEx){
Console.WriteLine ("httpEx Exception: " + httpEx.Message);
Console.WriteLine ("httpEx Inner Exception: " + httpEx.InnerException.Message);
JsonUser JsonUserError = new JsonUser ();
JsonUserError.ErrorMessage = "Error occured: " + httpEx.Message;
return JsonUserError;
}
}
When making a Web Request using ModernHttpClient, I generally follow the pattern below. Another great library created by Paul Betts is refit, and can be used to simplify rest calls.
using (var client = new HttpClient(new NativeMessageHandler(false, false)))
{
client.BaseAddress = new Uri(BaseUrl, UriKind.Absolute);
var result = await Refit.RestService.For<IRestApi>(client).GetData();
}
The second parameter for NativeMessageHandler should be set to true if using a customSSLVerification.
Here's a look at IRestApi
public interface IRestApi
{
[Get("/foo/bar")]
Task<Result> GetMovies();
}
Number of things I had to do to get this to work.
The Self Signed Cert had to allow TLS 1.2
As the API is Drupal based, HTTPS had to be enabled on the server and a module installed to manage the HTTP specific pages.