Is HTTPS Multipart file upload request a batch process? That is, if I have 99 files, and if the upload fails after file 95 has been uploaded, will that rollback the entire set?
I would assume that the entire set is rolled back, if you use plain HTTP(S) POST requests, as all uploads are POSTed in the same HTTP request. However, you could use JavaScript/AJAX to upload files individually, in a single request per file.
It depends what happens on the server.
If you have a single multipart request with 99 files, and as you read the stream, you save each file, then a failure on the 95th file will mean the first 94 files will have been processed and saved. If you need to rollback, you usually would have to implement that yourself.
If you have 95 files you must upload 1 by one. Because if we assume each file size is 1 MB. It would be a 95MB request. In order to perform the maximum successful operation, we should hit Apis multiple to avoid several errors like TimeOut Error, Stream too large error, etc.
Related
I have a scenario that
1. user upload a file - system process take 10mins -
2. then the user can transfer the file
I cannot do load test in a Thread - #1 then #2 because of the processing time (#2 is performing too quickly before the file is ready for transfer).
I am thinking to keep sending GET requests and finding the response containing the file name then proceeding to #2.
I can only use GET API to get a list of files (API does not support specific file search, the GET return list of file in response)
is that a good approach? how can I do it?
Use While Controller to wait until the uploaded file appears in the list of files from the API. The output of the API can be parsed using i.e. JSON Extractor
Check out Using the While Controller in JMeter for more comprehensive information and example configuration.
We're writing a Spring service that makes an HTTP endpoint available through which a video (or audio) file from an Amazon S3 store can be streamed. The basic idea is that you can type in an url in the Google Chrome address bar, and the service will fetch the file from S3 and stream it, in such a way that the user can start watching immediately without having to wait for a download to complete, and that the user can click on a random spot in the video's progress bar and immediately start watching the video from that spot.
The way I understand this should work in theory, is that Chrome starts downloading the file. The service responds with HTTP 200 and includes an Accept-Ranges: bytes and a Content-Length: filesize header. The filesize is known, because we can query that as metadata from S3 without fetching the entire file. Including these headers causes the browser to cancel the download, and request the file again with a Range: bytes=0-whatever header (where whatever is some chunk size that Chrome decides). The service then responds with HTTP 206 (Partial content) and the requested byte range, which we can determine easily because S3 supports the same range protocol. Chrome then requests successive chunks from the service, until the stream ends.
On the Spring side, we're sending the data out in a ResponseEntity<InputStreamResource> (as per this SO answer).
However, we observe in practice that while Chrome's cancels its first request after a few hundred bytes. However, it sends a second request with a Range: bytes=0- header, effectively asking for the entire file. The server responds with an HTTP 206. As a result, is has only downloaded a few hundred bytes of video, and the video obviously doesn't start playing.
Interestingly, in Firefox it all works properly. Unfortunately, our app needs to support Chrome. Are we missing some part of the protocol?
It turns out we had an off-by-one error in the Content-Range response header.
The syntax is Content-Range: bytes start-end/total. With a total of 10, if you want to get the entire range, you need to specify bytes 0-9/10, not 0-10/10, which was what we were doing.
Of course with the larger sizes of real files, and the actual ranges of chunks in the middle of such files, this error was a lot harder to notice than in the contrived example in the previous paragraph... ಠ_ಠ
Working on JMeter and trying to send the soap request to server and shows the below error msg.
Error Msg:- Cannot process the message because the content type 'application/soap+msbin1' was not the expected type 'application/xml; charset=utf-8'.
We need help to encode XML to 'application/soap+msbin1' format.
Bit late to the party, but I encountered a similar issue - I had a template for SOAP request which uses embedded-binary XML (xop:Include cid="...") and had to scratch my head to figure out how to do that with the stock HTTP Request.
The answer: you can't - not in a simple way. To solve the issue, I ended up customizing JMeter (I also looked at HTTPRawRequest as well but it doesn't seem to support https and I would have to rewrite a lot of the test script to use that). Since HTTP request does 99% of the job, the quickest way to support binary data is to change the source code to handle binary data.
The main issues are two: the Function interface in JMeter is designed around returning String, not byte[]. So already __FileToString() (which I used to read an external binary file to use) encodes the content of the file . Secondly, the HTTP Request Sampler and HTTPHC4Impl itself (excluding the "upload file" bit) encodes the parts of the HTTP request before sending it over to the wire.
Changing that implied changes in Function, AbstractFunction, CompoundVariable and create a new function class FileToStringBinary which encode the binary data in a way that it can be decoded after (by changes made to HTTPHC4Impl).
If I have the time I'll find someplace where to post the idea and the source (can't submit to JMeter because my update to HTTPHC4Impl is limited to handle the specific requests I need to test, where the embedded binary is in a multipart/related part, and I have no time or inclination to handle the general cases), but if you still need help to make it work, drop a line.
We need to call a third party lib, ideally at the time of onUploadChunk callback.
As shown in the documentation (http://docs.fineuploader.com/branch/master/api/events.html#uploadChunk), we can have some parameters in order to identify the chunk and do stuff with the javascript slice method.
But, the question is : how to give back updated chunk into the fineuploader upload process ?
Thanks a lot for help.
You cannot modify the chunks created by Fine Uploader, nor should you be able to as it may change the size of the chunk, the expected total number of chunks, and require adjustments of internal state and sent parameters. If you'd like to modify any files, you have two options:
Modify the file before it is sent to to Fine Uploader
Modify the file before the file upload begins. You can cancel the original file, and then submit the changed versions via the addFiles API method.
I want to design an api that allows clients to upload images, then the application creates different variants of the images, like resizing or changing the image format, finally the application stores the image information for each of the variants in a database. The problem occurs when I try to determine the proper strategy to implement this task, here are some different strategies i can think of.
Strategy 1:
Send a post request to /api/pictures/,
create all the image variants and return 201 created if all image files were created correctly and the image information was saved to the database, otherwise it returns a 500 error.
pros: easy to implement
cons: the client has to wait a very long time until all variants of the images are created.
Strategy 2:
Send a post request to /api/pictures/, create just the necessary information for the image variants and store it in the database, then returns a 202 accepted, and start creating the actual image variant files, the 202 response includes a location header with a new url, something like /api/pictures/:pictureId/status to 'monitor' the state of the image variants creation process. The client could use this url to check whether the process was completed or not, if the process was completed return a 201 created, if the process is pending return a 200 ok, if there is an error during the process, it ends and returns a 410 gone
pros: the client gets a very fast response, and it doesn't have to wait until all image variants are created.
cons: hard to implement server side logic, the client has to keep checking the returned location url in order to know when the process has finished.
Another problem is that, for example when the image variants are created correctly but one fails, the entire process returns a 410 gone, the client can keep sending requests to the status url because the application will try to create the failed image again, returning a 201 when its end correctly.
Strategy 3:
This is very similar to strategy 2 but instead of return a location for the whole 'process', it returns an array of locations with status urls for each image variant, this way the client can check the status for each individual image variant instead of the status of the whole process.
pros: same as strategy 2, if one image variant fails during creation, the other variants are not affected. For example, if one of the variants fails during creation it returns a 410 gone while the images that were created properly returns a 201 created.
cons: the client is hard to implement because it has to keep track of an array of locations instead of just one location, the number of requests increases proportionally to the number of variants.
My question is what is the best way to accomplish this task?
Your real problem is how to deal with asynchronous requests in HTTP. My approach to that problem is usually to adopt option 2, returning 202 Accepted and allowing the client to check current status with GET on the Location URI if he wants to.
Optionally, the client can provide a callback URI on a request header, which I will use to notify completion.