I've written a web API for sending and receiving data between a server and an app I'm working on.
It works fine for everything, but I've now found it won't let me send large strings. The strings are base64 strings which represent images, typically around 100kb in size.
The method I'm attempting is a multipart post, breaking the base64 string into chunks that can be sent successfully.
When I use this method, I get an error:
stream was reset: PROTOCOL_ERROR
Upon checking the database it seems the first string chunk sends successfully, but nothing more after that.
Can anyone shed some light on what's causing this to happen?
Relevant code is here:
First is the process for breaking the image into manageable chunks and posting it:
HTTPRequest req = new HTTPRequest();
IEnumerable<string> imgStrSplit = Split(img1byteArrayStr, 1000);
foreach (string s in imgStrSplit)
{
response = await req.SubmitImage("Test", s, "1");
}
And below is the SubmitImage() method of the HTTPRequest class:
public async Task<int> SubmitImage(string name, string imageString, string imgNum)
{
using (System.Net.Http.HttpClient client = new System.Net.Http.HttpClient(new NativeMessageHandler()))
{
client.DefaultRequestHeaders.Add("Accept", "application/json");
int successResponse;
string address = $"https://myURL/SubmitImage?name=" + name + "&imageStr=" + imageString + "&imgNum=" + imgNum + "&curl=AYZYBAYZE143";
HttpResponseMessage response = await client.GetAsync(address);
successResponse = JsonConvert.DeserializeObject<int>(response.Content.ReadAsStringAsync().Result);
return successResponse;
}
Thanks.
Found a solution via this discussion.
The problem was in the fact that I was using the ModernHTTPClint NuGet package, as seen in this line of the SubmitImage method:
using (System.Net.Http.HttpClient client = new System.Net.Http.HttpClient(new NativeMessageHandler()))
The NativeMessageHandler() is said to make connections work much faster, but apparently in some cases results in the PROTOCOL_ERROR I had encountered. By removing this part, I fixed the problem.
Related
I am from the old world that think webapi should return a strong typed object and let json serialization return data.
However, recently we got this requirement:
We have a sql table which has more than 500 columns.
The customer always want to return all the columns.
Our c# code does nothing other than reading the SqlDatareader, convert the reader to a c# object and return result.
In this case, wouldn't better to do this (example copied from another stackoverflow post). Basically just return a stream? Does returning a stream still considered to be anti-pattern?
public HttpResponseMessage SomeMethod(List<string> someIds)
{
HttpResponseMessage resp = new HttpResponseMessage();
resp.Content = new PushStreamContent(async (responseStream, content, context) =>
{
await CopyBinaryValueToResponseStream(responseStream, someIds);
});
return resp;
}
private static async Task CopyBinaryValueToResponseStream(Stream responseStream, int imageId)
{
// PushStreamContent requires the responseStream to be closed
// for signaling it that you have finished writing the response.
using (responseStream)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
await connection.OpenAsync();
using (SqlCommand command = new SqlCommand("SELECT 500 columns FROM [StupidWideTable] WHERE ....", connection))
{
.....
using (SqlDataReader reader = await command.ExecuteReaderAsync(CommandBehavior.SequentialAccess))
{
if (await reader.ReadAsync())
{
if (!(await reader.IsDBNullAsync(0)))
{
using (Stream data = reader.GetStream(0))
{
// Asynchronously copy the stream from the server to the response stream
await data.CopyToAsync(responseStream);
}
}
}
}
}
}
}// close response stream
}
Does returning a stream still considered to be anti-pattern?
Well, that depends on what you want to do. For example, if you want to return a 500 if the SQL server fails partway through, then you shouldn't return a stream.
Streaming results works fine on ASP.NET, but it's important to note that all headers (including the response status code) are sent before the stream begins. So you'll send an immediate 200 when you start streaming the result, and if there's an error later on there's no way to go back in time and change that to a 500. Or add some kind of Continue header.
In other words, yes it's supported; but you lose all the benefits of model binding, content negotiation, exception handlers, etc., because you're bypassing that whole pipeline.
I have a Spring boot API and one of the endpoints allows users to upload video's. Now My controller basically takes the file as a MultiPart file and then I store it in a temp folder accessible to tomcat. Once I have it stored on Disk, I then push the video to an S3 bucket.
Now to me anyway, this seems to be less than optimal, Like if I wanted to have a 100 or a 1000 users upload at once it seems really non performant to write the files to disk first.
As a little background I'm storing it on disk with the intention that if there is a issue pushing to S3 I can retry
The below code might show what I'm doing better than the above:
public Video addVideo(#RequestParam("title") String title,
#RequestParam("Description") String Description,
#RequestParam(value = "file", required = true) MultipartFile file) {
this.amazonS3ClientService.uploadFileToS3Bucket(file, title, description));
}
Method for storing Video file:
String fileNameWithExtenstion = awsS3FileName + "." + FilenameUtils.getExtension(multipartFile.getOriginalFilename());
//creating the file in the server (temporarily)
File file = new File(tomcatTempDir + fileNameWithExtenstion);FileOutputStream fos = new FileOutputStream(file);
fos.write(multipartFile.getBytes());
fos.close();PutObjectRequest putObjectRequest = new PutObjectRequest(this.awsS3Bucket, awsS3BucketFolder + UnigueId + "/" + fileNameWithExtenstion, file);
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
// Upload a file as a new object with ContentType and title
specified.amazonS3.putObject(putObjectRequest);
//removing the file created in the server
file.delete();
So my question is....is there a better way in Tomcat to:
A) Take in a file via a controllerB) Push to S3
There is no other way to do it with multipart. The problem with multipart that to properly segement parts from the requst they need sometimes skipped or be repeatable. That is impossible within memory w/o having memory to explode. Therefore, Commons FileUpload caches them on disk after a certain threshold is reached.
Multipart requests are the worst way for that. I highly recommend to use either PUT or POST with content type application/octet-stream. You can take the bare request input stream and pass to HttpClient to stream to your backend server. I did this already 5 years ago and it works for gigabytes. I have posted the solution in the Apache HttpClient mailing list.
There is one possibility how this could work under specific conditions:
All parts are in the correct physical order you want to read
Your write to a backend is fast enough to sustain the read from the front
Consume the root part and then go over to the next physical one, process the request body lazily. JAX-WS RI (Metro) has a very nice handling of multipart requests for XOP/MTOM. Learn from that because you won't be able to make it any better.
Perhaps you can try to direct stream the input stream from your MultipartFile to S3.
Consider the following uploadFileToS3Bucket method:
public PutObjectResult uploadFileToS3Bucket(InputStream input, long size, String title, String description) {
// Indicate the length of the information to avoid the need to compute it by the AWS SDK
// See: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html#PutObjectRequest-java.lang.String-java.lang.String-java.io.InputStream-com.amazonaws.services.s3.model.ObjectMetadata-
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(size); // rely on Spring implementation. Maybe you probably also can use input.available()
// compute the object name as appropriate
String key = "...";
PutObjectRequest putObjectRequest = new PutObjectRequest(
this.awsS3Bucket, key, input, objectMetadata
);
// The rest of your code
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
// Upload a file as a new object with ContentType and title
return specified.amazonS3.putObject(putObjectRequest);
}
Of course, you need to provide the service the input stream obtained from the client request associated with the MutipartFile object:
public Video addVideo(
#RequestParam("title") String title,
#RequestParam("Description") String Description,
#RequestParam(value = "file", required = true) MultipartFile file) {
try (InputStream input = file.getInputStream()) {
this.amazonS3ClientService.uploadFileToS3Bucket(input, file.getSize(), title, description));
}
}
Probably you can also play with the getBytes method of MultipartFile and create a ByteArrayInputStream to perform the operation.
In addVideo:
byte[] bytes = file.getBytes();
In uploadFileToS3Bucket:
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(bytes.length);
PutObjectRequest putObjectRequest = new PutObjectRequest(
this.awsS3Bucket, key, new ByteArrayInputStream(bytes), objectMetadata
);
I would prefer the first solution, but try to determine which option offers you the best performance.
I'm using HttpSelfHostConfiguration to create a WebAPI (service). My goal is to have one route stream the mjpeg video from a security feed and have other routes available for configuration and the web interface.
The issue I'm having is that every example I've come across expects a known quantity of images in order to set the content-length for the main response. I don't have this, and flushing the stream doesn't do anything.
Here's the current code for the response. If I use this same code with raw sockets instead of through an ApiController, I can stream it just fine, but creating a webserver from scratch for everything else I'm needing doesn't seem like a lot of fun.
[HttpGet]
public HttpResponseMessage Stream(int channel)
{
var response = Request.CreateResponse();
response.Content = new PushStreamContent((outputStream, content, context) =>
{
StreamWriter writer = new StreamWriter(outputStream);
while (true)
{
using (MemoryStream ms = new MemoryStream())
{
ReadMemoryMappedFile(channel);
ms.SetLength(0);
this.Image.Bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
byte[] buffer = ms.GetBuffer();
writer.WriteLine("--boundary");
writer.WriteLine("Content-Type: image/jpeg");
writer.WriteLine(string.Format("Content-length: {0}", buffer.Length));
writer.WriteLine();
writer.Write(buffer);
writer.Flush();
}
}
});
response.Content.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("multipart/x-mixed-replace; boundary=--boundary");
return response;
}
I hope my late answer will help, since I ran into the same issue recently and it took me some time to figure it out...
My solution was to specify the boudary int the ContentType without the "--" (but you need to keep them while writing in the stream).
Try to configure the Headers like that :
response.Content.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("multipart/x-mixed-replace; boundary=boundary");
And write boundaries in the stream like that :
writer.WriteLine("--boundary");
Like this it works for me.
I couldn't find anywhere that explicitly states this, but I'm going to assume that HttpSelfHostConfiguration does not support the functionality I'm looking for, and always requires the stream to be closed before it will release the buffer.
I swapped HttpSelfHostConfiguration with OWIN.SelfHost and it works as expected.
I have an endpoint that needs to accept a file upload and also some other information from the client request. With the following code I can upload the file successfully but can't seem to figure out how to read the other info.
I make a test request from Postman with the following form data:
image -- myimage.jpg -- of type File
email -- a#b.com -- of type Text
The backend code looks like this:
[HttpPost]
public async Task<HttpResponseMessage> SharePhoto()
{
try
{
var provider = new MultipartMemoryStreamProvider();
var data = await Request.Content.ReadAsMultipartAsync(provider);
// this is how I get the image which I am succesfully passing to EmailService
var item = (StreamContent)provider.Contents[0];
using (var stream = new MemoryStream())
{
await item.CopyToAsync(stream);
String emailAddress;
EmailService.SendSharedPhoto(emailAddress, stream);
return Request.CreateResponse();
}
}
catch
{
// do stuff
}
}
In this example I am able to access provider.Contents[1] but can't seem to be able to get the value from it into emailAddress. I'm thinking it may be possible to use the same trick as the await item.CopyToASync(stream) from the image upload, but I'm hoping I can get a simpler solution to that. Any ideas?
I just barely answered a very similar question to this yesterday. See my answer here complete with sample controller code.
The method I ended up using is:
If the form elements are strings (and it worked for me since the mobiel frontend took responsability for input data) you can do this:
var streamContent = (StreamContent)provider.Contents[1];
var memStream = new MemoryStream();
await streamContent.CopyToAsync(memStream);
var actualString = Encoding.UTF8.GetString(x.ToArray());
If however the field needs to represent a collection of items, like for example the email list: ["a#b.com", "x#c.com"], etc a JavaScriptSerializer can be user, like so:
var streamContent = (StreamContent)provider.Contents[1];
var emailAddresses = await str.ReadAsStringAsync();
var jsSerializer = new JavaScriptSerializer();
var deserializedData = jsSerializer.Deserialize<string[]>(emailAddresses);
Note that this is nowhere near safe, though it is few lines of code and happens to work.
In our app (Silverlight 5 out-of-browser client hitting a WebApi server) we routinely use an HttpClient for posting/getting/deleting and so on all our entities between client and server. This all works fine most of the time, but recently we have run into an issue when uploading (posting) larger entities (> 30/35mb): we start the streaming process and BEFORE it is finished our Post method on the Web API is hit, receiving a null entity.
We can't understand what is going on, and suspect there must be some timing issue related since it all depends on the size of the upload.
To further explain, our client in summary is doing this:
HttpResponseMessage response = await _client.SendAsync(request);
string jsonResult = await response.Content.ReadAsStringAsync();
... where _client is our HttpClient and request our HttpRequestMessage. In case it is also relevant (I am trying not to flood the question with code :), the content in the request is created like this:
request.Content = new StringContent(JsonConvert.SerializeObject(content), Encoding.UTF8, "application/json");
Well, when we debug this the Post method on our server is hit before the await _client.SendAsync(request) finishes, which sort of "explains" why it is receiving a null entity in such cases (larger entities), where when it works that await call is finished and THEN the Post is hit.
In case if sheds more light into it, due to certain limitations on the HttpClient (regarding access to AllowWriteStreamBuffering), we have also tested an equivalent scenario but using directly an HttpWebRequest... unfortunately, the behavior is exactly the same. This is the relevant extract:
httpRequest.BeginGetRequestStream(RequestStreamCallback, httpRequest);
(where httpRequest is our HttpWebRequest with AllowWriteStreamBuffering = false), and the callback to handle the request stream is as follows:
private void RequestStreamCallback(IAsyncResult ar)
{
var request = ar.AsyncState as System.Net.HttpWebRequest;
if (request != null)
{
var requestStream = request.EndGetRequestStream(ar);
var streamWriter = new StreamWriter(requestStream) {AutoFlush = true};
streamWriter.Write(_jsonContent);
streamWriter.Close();
requestStream.Close(); // Belt and suspenders... shouldn't be needed
// Make async call for response
request.BeginGetResponse(ResponseCallback, request);
}
}
Again, for larger entities when we debug the Post method on the Web API is hit (with a null parameter) BEFORE the streamWriter.Write finalizes and the streamWriter.Close is hit.
We've been reading all over the place and fighting with this for days on now. Any help will be greatly appreciated!
In case somebody runs into this, I finally figured out what was going on.
In essence, the model binding mechanism in the Web API Post method was throwing an exception when de-serializing the JSON, but the exception was somewhat "hidden"... at least if you did not know that much about the inner workings of the Web API, as was my case.
My Post method originally lacked this validation check:
var errors = "";
if (!ModelState.IsValid)
{
foreach (var prop in ModelState.Values)
{
foreach (var modelError in prop.Errors.Where(modelError => modelError != null))
{
if (modelError.Exception != null)
{
errors += "Exception message: " + modelError.Exception.Message + Environment.NewLine;
errors += "Exception strack trace: " + modelError.Exception.StackTrace + Environment.NewLine;
}
else
errors += modelError.ErrorMessage + Environment.NewLine;
errors += " --------------------- " + Environment.NewLine + Environment.NewLine;
}
}
return Request.CreateErrorResponse(HttpStatusCode.NoContent, errors);
}
This is a "sample" check, the main idea being verifying the validity of the ModelState... in our breaking scenarios is wasn't valid because the Web API hadn't been able to bind the entity, and the reason could be found within the Errors properties of the ModelState.Values. The Post was being hit ok, but with a null entity, as mentioned.
By the way, the problem was mainly caused by the fact that we weren't really streaming the content, but using a StringContent which was attempted to be de-serialized in full... but that is another story, we were mainly concerned here with not understanding what was breaking and where.
Hope this helps.