Video - stream - this works.. however on larger files it is slow..
How can I improve the content s3 store to deliver the content faster ?
I have tried returning a byte array and copy to a buffer.. all load.. just slow... I am not sure where the bottle neck is coming from..
Optional<File> f = filesRepo.findById(id);
if (f.isPresent()) {
InputStreamResource inputStreamResource = new InputStreamResource(contentStore.getContent(f.get()));
HttpHeaders headers = new HttpHeaders();
headers.setContentLength(f.get().getContentLength());
headers.set("Content-Type", f.get().getMimeType());
return new ResponseEntity<Object>(inputStreamResource, headers, HttpStatus.OK);
}
also get this warning:
Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
Related
I'm trying to stream Mp4 videos on my website, and support Range so you can easily navigate large videos without having to download the entire thing. Sadly for whatever reason sometimes the entire video up to the point you select is downloaded. It's like Range works half of the time
#GetMapping(value = "/videos/{fileName}", produces = "video/mp4")
public ResponseEntity<FileSystemResource> streamFile(
#PathVariable("fileName") String fileName) throws MalformedURLException {
String clipid = fileName.split("\\.")[0];
final Video video = videoService.getVideo(clipid.trim());
if (video == null) {
return ResponseEntity.status(HttpStatus.NOT_FOUND).body(null);
}
return ResponseEntity.ok().body(new FileSystemResource(video.getFile()));
}
The responses from the server:
First position move, GOOD:
Request: range: bytes=218202112-
Response: Content-Range: bytes 218202112-596696593/596696594
Second attempt to move the time position, Full download:
Request: range: bytes=365658112-
Response: No content Range, only length
So strangely enough, the issue was brought on by Cloudflare's cache system.
To fix it I simply created a page-rule to bypass caching on my video endpoints.
I upload a 3.5 MB file from my MVC app. When I go to send that file to my web api endpoint, I notice the request's content-length is double the size of the file (7 MB).
I tested this theory with a 5 MB file and sure enough the content-length when I went to send to the web api was 10 MB.
Below is how I am sending the file to my web api endpoint:
using (HttpClient client = new HttpClient())
{
client.BaseAddress = new Uri(url);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
return await client.PostAsync(requestUri, new StringContent(serializedContent, Encoding.Unicode, "application/json"));
}
I am calling this method from my MVC controller in the POST method. Why does my content-length get doubled?
UPDATE:
I should note that I am using JSON.NET's JsonConvert.SerializeObject method to convert the object that contains the byte array to a string
You're using Encoding.Unicode which uses 16-bit characters by default. If you want to save roughly half the space then use Encoding.UTF8 which uses 8-bit characters by default. Note, characters that can't be expressed in just 8-bits will use multiple bytes.
I am using the Java Amazon AWS SDK to perform some multipart uploads from HDFS to S3. My code is the following:
for (int i = startingPart; currentFilePosition < contentLength ; i++)
{
FSDataInputStream inputStream = fs.open(new Path(hdfsFullPath));
// Last part can be less than 5 MB. Adjust part size.
partSize = Math.min(partSize, (contentLength - currentFilePosition));
// Create request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucket).withKey(s3Name)
.withUploadId(currentUploadId)
.withPartNumber(i)
.withFileOffset(currentFilePosition)
.withInputStream(inputStream)
.withPartSize(partSize);
// Upload part and add response to our list.
partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
currentFilePosition += partSize;
inputStream.close();
lastFilePosition = currentFilePosition;
}
However, the uploaded file is not the same as the original one. More specifically, I am testing on a test file, which has about 20 MB. The parts I upload are 5 MB each. At the end of each 5MB part, I see some extra text, which is always 96 characters long.
Even stranger, if I add something stupid to .withFileOffset(), for example,
.withFileOffset(currentFilePosition-34)
the error stays the same. I was expecting to get other characters, but I am getting the EXACT 96 extra characters as if I hadn't modified the line.
Any ideas what might be wrong?
Thanks,
Serban
I figured it out. This came from a stupid assumption on my part. It turns out, the file offset in ".withFileOffset(...)" tells you the offset where to write in the destination file. It doesn't say anything about the source. By opening and closing the stream repeatedly, I am always writing from the beginning of the file, but to a different offset. The solution is to add a seek statement after opening the stream:
FSDataInputStream inputStream = fs.open(new Path(hdfsFullPath));
inputStream.seek(currentFilePosition);
I want to write the response on an HTTP request to a File. However I want to stream the response to a physical file without waiting for the entire response to be loaded.
I will actually be making a request to a JHAT server for returning all the Strings from the dump. My browser hangs before the response completes as there are 70k such objects, I wanted to write them to a file so that I can scan through.
thanks in advance,
Read a limited amount of data from the HTTP stream and write it to a file stream. Do this until all data has been handled.
Here is example code demonstrating the principle. In this example I do not deal with any i/o errors. I chose an 8KB buffer to be faster than processing one byte at a time, yet still limiting the amount of data pulled into RAM during each iteration.
final URL url = new URL("http://example.com/");
final InputStream istream = url.openStream();
final OutputStream ostream = new FileOutputStream("/tmp/data.txt");
final byte[] buffer = new byte[1024*8];
while (true) {
final int len = istream.read(buffer);
if (len <= 0) {
break;
}
ostream.write(buffer, 0, len);
}
I'm trying to read Shoutcast stream and then play it using MediaStreamSource. Here is the excellent open source project that saved lot of my time. After little bit modification I'm able to hear perfect sound. But the problem is I'm getting a periodic blip/hiccup kind of noise.
Any idea how I can stop that noise. I thought its may be Shoutcast sends some metadata in interval but don't find out how to stop that. Tried with request.Headers["Icy-MetaData"] = "0";
But it doesn't fix my problem either. Any help will be greatly appreciated.
Edit1:
I did some more investigation. I read my stream data for 2-3 mins and found that there are lot of 'zero' byte in that stream. Here is the list of index of '0' byte
92 247 359 1208 1904 2037 2227 2397 2536 2694 2740 2863 2952 3048 3110 3689 3994 4027 4098 4218 4730 4830 4943 5029 5115 5248 5315 5358 5666 6084 6375 6873 6920 7441 7660 7700 7756 8174 8254 8614 9010 9018 9025 9039 9541 9846.....
Is it because httpwebrequest slow download/failed to download or Shoutcast itself sending those zero bytes? Also does this '0' bytes causing that hiccup noise?
Edit2:
Here is few line of code of how I'm getting response from shoutcast
HttpWebRequest request = result.AsyncState as HttpWebRequest;
HttpWebResponse response = request.EndGetResponse(result) as HttpWebResponse;
r = response.GetResponseStream();
ShoutcastHeader(r);
And here is my ShoutcastHeader method definition:
StreamReader headerReader = new StreamReader(r);
bool headerIsDone = false;
while (!headerIsDone)
{
string headerLine = headerReader.ReadLine();
if (headerLine.StartsWith("icy-name:"))
{
StationName = headerLine.Substring(9);
}
else if (headerLine.StartsWith("icy-genre:"))
{
Genre = headerLine.Substring(10);
}
else if (headerLine.StartsWith("icy-br:"))
{
BitRate = short.Parse(headerLine.Substring(7));
}
else if (headerLine.StartsWith("icy-metaint:"))
{
MetaInt = int.Parse(headerLine.Substring(12)) * 1111084;
MetadataAvailable = true;
}
else if (headerLine.Equals(""))
headerIsDone = true;
}
And here is the response in headerReader
ICY 200 OK
icy-notice1:This stream requires Winamp
icy-notice2:SHOUTcast Distributed Network Audio Server/Linux v1.9.93atdn
icy-name:Bollywood & Beyond - Radio NRI 24/7
icy-genre:Indian Hindi Tamil Telugu Malayalam Desi
icy-url:http://www.radionri.com
content-type:audio/mpeg
icy-pub:1
icy-br:128
Also I have places the stream bytes in my skydrive share location.
Based on this thread it looks like the hiccups are the header information being resent periodically. There is an issue on ShoutStreamSource project referencing this too:
"The current implementation does not discard the mp3headers after the first one has been parsed. This causes hicups in the sound coming from the stream, and will be fixed in the near future."
However at the time of this answer it looks like the project was put on hold. Hopefully you or someone else can add this functionality. At least knowing what the problem is should help with a solution.