Multipart request rejected because no bounday were found while boundary is already sent - spring-boot

I went through the other similar questions but couldn't find any solution. We have a backend service running by Spring Boot and has been working for a while now. But there is a new user of this service recently and they are using MuleSoft to send their request. But all attempts to send a file to our service fails with this error:
Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: the request was rejected because no multipart boundary was found"
The only difference we could find between a request from MuleSoft and say a curl command is that MuleSoft always sends the request with boundary value wrapped with double quotes
Mule request:
<Header name="Content-Type">multipart/form-data; charset=UTF-8; boundary="--------------------------669816398596264398718285"</Header>
Versus Postman/curl request:
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd656810a00)
> POST /api/upload HTTP/2
> Host: myhost
> user-agent: curl/7.79.1
> accept: */*
> content-length: 97255
> content-type: multipart/form-data; boundary=------------------------111fd08cb4fafe1c
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 200
< date: Mon, 19 Dec 2022 04:56:25 GMT
< content-length: 0
Our controller in Spring is very simple:
#RestController
class MyController {
#PostMapping("/upload", consumes = [MediaType.MULTIPART_FORM_DATA_VALUE])
#ResponseStatus(HttpStatus.OK)
fun uploadDocument(#RequestPart("file") file: MultipartFile) {
logger.info { "ContentType: ${file.contentType}" }
logger.info { "Name: ${file.name}" }
logger.info { "Byte: ${String(file.bytes)}" }
}
}
The following curl command works fine:
curl -v -X POST -F file=#/Users/myhomefolder/Documents/some-file.jpg https://host-name/api/upload
But this script from MuleSoft doesn't (Sorry I'm not familiar with Mule, I got this code from their team):
import dw::module::Multipart
output multipart/form-data boundary = "---WebKitFormBoundary7MA4YWxkTrZu0gW"
---
{
parts : {
file : {
headers : {
"Content-Disposition" : {
"name": "file",
"filename": payload.name
},
"Content-Type" : "multipart/form-data"
},
content : payload.byteArray
}
}
}
Is there any configuration in Spring that accepts double quotes for boundary? Is there anything missing in our backend configuration that should be added to support different HTTP client?

Related

How to send file with curl to a spring-boot-starter-webflux app?

I'm trying to send a file to a spring-boot-starter-webflux application.
This is my controller:
#RestController
#RequestMapping("/fplan")
public class FplanController {
private final FplanService fplanService;
public FplanController(FplanService fplanService) {
this.fplanService = fplanService;
}
#PostMapping(value = "/file")
public Flux<Boolean> handleFileUpload(#RequestPart("file") MultipartFile file) throws IOException {
LOGGER.info(file.getOriginalFilename());
return fplanService.newFplan(file.getInputStream());
}
}
And this is my curl command line:
curl -v -F 'file=#fplan.txt' 'http://localhost:8082/fplan/file'
And this is the error output:
* Trying 127.0.0.1:8082...
* Connected to localhost (127.0.0.1) port 8082 (#0)
> POST /fplan/file HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 16985001
> Content-Type: multipart/form-data; boundary=------------------------76d46a224dfc0ceb
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 415 Unsupported Media Type
< Content-Type: application/json
< Content-Length: 137
<
{"timestamp":"2022-03-07T15:30:53.056+00:00","path":"/fplan/file","status":415,"error":"Unsupported Media Type","requestId":"936a38c5-5"}
I already tried:
#PostMapping(value = "/file", consumes = MediaType.ALL_VALUE)
or
#PostMapping(value = "/file", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
or
curl -v -X POST -H "Content-Type: multipart/form-data" -F 'file=#./fplandb_fv.txt;type=text/plain' "http://localhost:8082/fplan/file"
without success.
Any ideas whats missing?
After all I found this post POST binary file with cmd line curl using headers contained in the file.
If I change the controller method like this
#PostMapping(value = "/file", consumes = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public Flux<Boolean> handleFileUpload(#RequestBody byte[] bytes) throws IOException {
LOGGER.info("Received {} bytes", bytes.length);
return fplanService.newFplan(new ByteArrayInputStream(bytes));
}
then this curl line sends the file:
curl -v --header "Content-Type:application/octet-stream" --data-binary #fplandb_fv.txt 'http://localhost:8082/fplan/file'
Feel free to post an answer that explains how to send a file for the controller implementation in the question.

Kotlin & Spring MVC - HTTP Status 400 - Null

I have a Kotlin #RestController, and I would expect it to return 400 Bad Request in a situation where a null is passed for a #RequestParam argument.
Example:
#RestController
class Api() {
#PostMapping("endpoint")
fun endpoint(#DateTimeFormat(iso = DATE) #RequestParam date: LocalDate) {
//do something
}
}
If I were to make a request to POST /endpoint?date I get a 500 Internal Server Error with the following (shortened) body:
{
"timestamp": "2020-09-14T20:39:38.102+0000",
"status": 500,
"error": "Internal Server Error",
"message": "Parameter specified as non-null is null: method Api.endpoint, parameter date",
"trace": "java.lang.IllegalArgumentException: Parameter specified as non-null is null: method Api.endpoint, parameter date\r\n\tat Api.endpoint(Api.kt)\r\n\t
...
...
atjava.base/java.lang.Thread.run(Thread.java:834)\r\n",
"path": "/campaigns/contributions/unfunded/retries"
}
Is there any way to fix this using either some additional library, a configuration, or some custom code that does not have other side effects that keeps everything the same except that the status code will be 400 Bad Request
The following works for Kotlin 1.4.*. This is not necessarily the best answer because there is no guarantee that Kotlin will not change the excption type of message in future releases. For eaxample, when I originally asked the question I was using 1.3.*, and the exception was IllegalArgumentException. In 1.4.* it has been changed to NullPointerException.
#ControllerAdvice
class KotlinNonNullParamHandling {
#ExceptionHandler
protected fun handleKotlinNonNullViolation(
exception: NullPointerException,
response: HttpServletResponse
) {
val nullParameter = exception.message?.startsWith("Parameter specified as non-null is null: ") == true
val restController = exception
.stackTrace
.getOrNull(0)
?.let { Class.forName(it.className) }
?.getAnnotation(RestController::class.java) != null
val status =
if (nullParameter && restController) HttpStatus.BAD_REQUEST
else HttpStatus.INTERNAL_SERVER_ERROR
response.sendError(status.value(), exception.message)
}
}
Before assuming that it is 400, we are checking that the exception
is a NullPointerException
has a message starting with "Parameter specified as non-null is null: "
it was thrown from a class the is RestController (otherwise it might be some other problem, such as a null being sent to a method by reflection having nothing to do with the web layer.)
Spring will throw the IllegalArgumentException by default when a validation fails here. Every exception inside your Spring-Boot application will be treated by default as InternalServerError.
You can modify your response code by adding a function like this:
#RestController
class Api() {
#ExceptionHandler(IllegalArgumentException.class)
#ResponseStatus(HttpStatus.BAD_REQUEST)
public void onIllegalArgumentException(IllegalArgumentException e) {
}
#PostMapping("endpoint")
fun endpoint(#DateTimeFormat(iso = DATE) #RequestParam date: LocalDate) {
//do something
}
}
Afterwards you'll get the 400 status code on invalid request param.
I've tested your code, and there's nothing obviously wrong with it - works as expected.
curl -v -d "date=2020-10-01" http://localhost:8080/endpoint
* Trying ::1:8080...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /endpoint HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 15
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 15 out of 15 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Content-Length: 0
< Date: Tue, 13 Oct 2020 07:04:21 GMT
<
* Connection #0 to host localhost left intact
So I would:
Verify that the client is properly formatting the request - e.g. try explicitly setting the Content-Type header to application/x-www-form-urlencoded, check if there's no typo in the param name etc.
That you have Jackson properly configured to work with Kotlin (com.fasterxml.jackson.module:jackson-module-kotlin - dependency)

Spring Boot Validation of Path Variables returns blank message

I'm using spring boot validation to validate the #PathVariable as shown in the code below, the validation works however there is no message returned when I execute through curl.exe, I expect to receive the default message.
#GetMapping("/number/{id}")
String getNumber(#PathVariable #Min(2) Long id) {
System.out.println("getNumber :" + id);
return "param id is : " + id;
}
I have specified a CustomGlobalExceptionHandler to catch ConstraintViolationException and this works in that the returned http status code is 400 and it displays a message in the console log "must be greater than or equal to 2"
#ExceptionHandler(ConstraintViolationException.class)
#ResponseStatus(HttpStatus.BAD_REQUEST)
#ResponseBody
void onConstraintValidationException(ConstraintViolationException e) {
System.out.println("in onConstraintValidationException - start");
String error = "blank";
for (ConstraintViolation<?> violation : e.getConstraintViolations()) {
error = violation.getMessage();
System.out.println(error);
}
System.out.println("in onConstraintValidationException - end");
}
I'm using the command line to execute the REST Service
curl.exe -v -X GET "http://localhost:8080/demo-0.0.2-SNAPSHOT/number/1" -H "accept: /"
and I receive this output, but no message
*Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /demo-0.0.2-SNAPSHOT/number/1 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.55.1
> accept: */*
>
< HTTP/1.1 400
< Transfer-Encoding: chunked
< Date: Tue, 11 Aug 2020 09:28:06 GMT
< Connection: close
<
* Closing connection 0*
I based this on the mykong.com example https://mkyong.com/spring-boot/spring-rest-validation-example/ section 3. Path Variables Validation. where the results should match
curl -v localhost:8080/books/0
{
"timestamp":"2019-02-20T13:35:59.808+0000",
"status":400,
"error":"Bad Request",
"message":"findOne.id: must be greater than or equal to 1",
"path":"/books/0"
}
Can you suggest why I don't receive a message returned back to the command line?
Did you add these annotations to your controller:
import org.springframework.validation.annotation.Validated;
import javax.validation.constraints.Min;
#RestController
#Validated
You have to implement this class in order to get 400 message instead of 500:
#ControllerAdvice
public class CustomGlobalExceptionHandler extends ResponseEntityExceptionHandler {
#ExceptionHandler(ConstraintViolationException.class)
public void constraintViolationException(HttpServletResponse response) throws IOException {
response.sendError(HttpStatus.BAD_REQUEST.value());
}
//..
}

Sending .gz file via CURL to RESTful put creating ZipException in GZIPInputStream

The application I am creating takes a gzipped file sent to a RESTful PUT, unzips the file and then does further processing like so:
public class Service {
#PUT
#Path("/{filename}")
Response doPut(#Context HttpServletRequest request,
#PathParam("filename") String filename,
InputStream inputStream) {
try {
GZIPInputStream gzipInputStream = new GZIPInputStream(inputStream);
// Do Stuff with GZIPInputStream
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}
I am able to successfully send a gzipped file in a unit test like so:
InputStream inputStream = new FileInputStream("src/main/resources/testFile.gz);
Service service = new Service();
service.doPut(mockHttpServletRequest, "testFile.gz", inputStream);
// Verify processing stuff happens
But when I build the application and attempt to CURL the same file from the src/main/resources dir with the following I get a ZipException:
curl -v -k -X PUT --user USER:Password -H "Content-Type: application/gzip" --data-binary #testFile.gz https://myapp.dev.com/testFile.gz
The exception is:
java.util.zip.ZipException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
at Service.doPut(Service.java:23)
// etc.
So does anyone have any idea why sending the file via CURL causes the ZipException?
Update:
I ended up taking a look at the actual bytes being sent via the InputStream and figured out where the ZipException: Not in GZIP format error was coming from. The first two bytes of a GZIP file are required to be 1F and 8B respectively in order for GZIPInputStream to recognize the data as being in GZIP format. Instead the 8B byte, along with every other byte in the steam that doesn't correspond to a valid UTF-8 character, was transformed into the bytes EF, BF, BD which are the UTF-8 unknown character replacement bytes. Thus the server is reading the GZIP data as UTF-8 characters rather than as binary and is corrupting the data.
The issue I am having now is I can't figure out where I need to change the configuration in order to get the server to treat the compressed data as binary vs UTF-8. The application uses Jax-rs on a Jersey server using Spring-Boot that is deployed in a Kubernetes pod and ran as a service, so something in the setup of one of those technologies needs to be tweaked to prevent improper encoding from being used on the data.
I have tried adding -H "Content-Encoding: gzip" to the curl command, registering the EncodingFilter.class and GZipEncoder.class in jersey ResourceConfig class, adding application/gzip to the server.compression.mime-types in application.propertes, adding the #Consumes("application/gzip") annotation to the doPut method above, and several other things I can't remember off the top of my head but nothing seems to have any effect.
I am seeing the following in the verbose CURL logs:
> PUT /src/main/resources/testFile.gz
> HOST: my.host.com
> Authorization: Basic <authorization stuff>
> User-Agent: curl/7.54.1
> Accept: */*
> Content-Encoding: gzip
> Content-Type: application/gzip
> Content-Length: 31
>
} [31 bytes data]
* upload completely sent off: 31 out of 31 bytes
< HTTP/1.1 500
< X-Application-Context: application
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: <date stuff>
...etc
Nothing I have done has affected the receiving side
Content-Type: application/json;charset=UTF-8
portion, which I suspect is the issue.
I met the same problem and finally solved it by using -H 'Content-Type:application/json;charset=UTF-8'
Use Charles to find the difference
I can successfully send the gzipped file using Postman. So I used Charles to catch two packages sent by curl and postman respectively. After I compared these two packages, I found that Postman used application/json as Content Type while curl used text/plain.
Spring docs: Content Type and Transformation
According to Spring docs, if the content type is text/plain and the source payload is byte[], Spring will convert the payload to string using charset specified in the content-type header. That's why ZipException occurred. Since the original byte data had already been decoded and not in gzip format anymore.
Spring source code
#Override
protected Object convertFromInternal(Message<?> message, Class<?> targetClass, #Nullable Object conversionHint) {
Charset charset = getContentTypeCharset(getMimeType(message.getHeaders()));
Object payload = message.getPayload();
return (payload instanceof String ? payload : new String((byte[]) payload, charset));
}

CXF JAX-RS client always sends empty PUT requests in chunking mode regardles of AllowChunking setting

We perform PUT request to our party using CXF JAX-RS client. Request body is empty.
A simple request invocation leads to server response with code 411.
Response-Code: 411
"Content-Length is missing"
Our party's REST-server requires Content-Length HTTP-header to be set.
We switched chunking off according to note about chunking but this did not solve the problem. The REST-server still answers with 411 error.
Here is our conduit configuration from cxf.xml file
<http-conf:conduit name="{http://myhost.com/ChangePassword}WebClient.http-conduit">
<http-conf:client AllowChunking="false"/>
</http-conf:conduit>
Line in the log confirms that execution of our request bound to our conduit configuration:
DEBUG o.a.cxf.transport.http.HTTPConduit - Conduit '{http://myhost.com/ChangePassword}WebClient.http-conduit' has been configured for plain http.
Adding Content-Length header explicitly also did not help.
Invocation.Builder builder = ...
builder = builder.header(HttpHeaders.CONTENT_LENGTH, 0);
A CXF Client's log entry confirms header setting, however when we sniffed packets, we have surprisingly found that header setting has been completely ignored by CXF client. Content-Length header was not sent.
Here is the log. Content-Length header is present:
INFO o.a.c.i.LoggingOutInterceptor - Outbound Message
---------------------------
ID: 1
Address: http://myhost.com/ChangePassword?username=abc%40gmail.com&oldPassword=qwerty123&newPassword=321ytrewq
Http-Method: PUT
Content-Type: application/x-www-form-urlencoded
Headers: {Accept=[application/json], client_id=[abcdefg1234567890abcdefg12345678], Content-Length=[0], Content-Type=[application/x-www-form-urlencoded], Cache-Control=[no-cache], Connection=[Keep-Alive]}
--------------------------------------
DEBUG o.apache.cxf.transport.http.Headers - Accept: application/json
DEBUG o.apache.cxf.transport.http.Headers - client_id: abcdefg1234567890abcdefg12345678
DEBUG o.apache.cxf.transport.http.Headers - Content-Length: 0
DEBUG o.apache.cxf.transport.http.Headers - Content-Type: application/x-www-form-urlencoded
DEBUG o.apache.cxf.transport.http.Headers - Cache-Control: no-cache
DEBUG o.apache.cxf.transport.http.Headers - Connection: Keep-Alive
And here is an output of the packet sniffer. Content-Length header is not present:
PUT http://myhost.com/ChangePassword?username=abc%40gmail.com&oldPassword=qwerty123&newPassword=321ytrewq HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
client_id: abcdefg1234567890abcdefg12345678
Cache-Control: no-cache
User-Agent: Apache-CXF/3.1.8
Pragma: no-cache
Host: myhost.com
Proxy-Connection: keep-alive
Does anyone know how actually disable chunking?
Here is our code:
public static void main(String[] args)
{
String clientId = "abcdefg1234567890abcdefg12345678";
String uri = "http://myhost.com";
String user = "abc#gmail.com";
Client client = ClientBuilder.newBuilder().newClient();
WebTarget target = client.target(uri);
target = target.path("ChangePassword").queryParam("username", user).queryParam("oldPassword", "qwerty123").queryParam("newPassword", "321ytrewq");
Invocation.Builder builder = target.request("application/json").header("client_id", clientId).header(HttpHeaders.CONTENT_LENGTH, 0);
Response response = builder.put(Entity.form(new Form()));
String body = response.readEntity(String.class);
System.out.println(body);
}
Versions:
OS: Windows 7 Enterprise SP1
Arch: x86_64
Java: 1.7.0_80
CXF: 3.1.8
I had a very similar issue that I was not able to solve as you did by trying to turn off chunking.
What I ended up doing was setting the Content-Length to 1 and adding some white space " " as the body. For me it seemed that the proxy servers before the server application was rejected the request and by doing that got me past the proxy servers and the server was able to process the request as it was only operating based on the URL.

Resources