I'm working on a class to download videos from an url.
I want to stream these videos instead of downloading them at once, so my program uses less RAM.
The function is the following
def get_file(url, max_segment_size)
http_client = HTTPClient.new
segment = nil
http_client.get_content(url) do |chunk|
segment.nil? ? segment = chunk : segment << chunk
if segment.size >= max_segment_size
# send part to s3
send_part(segment)
segment = nil
end
end
# send last part
send_part(segment) if segment
end
However, the program still uses a lot of RAM. For example, streaming a file of 30MB makes the process consume 150MB. Comparing to downloading the whole file at once, it uses about the same amount of ram. (I tried using net/http with the read_body method. Same results)
My understanding was that setting segment = nil should free up the space on the memory that the variable was using.
Is this expected to happen? Is there a way to manually free up this space on ruby?
I'm currently implementing offline streaming with FairPlay streaming. Therefor I'm downloading streams using an AVAssetDownloadTask.
I want to give the users feedback about the size of the download which starts to begin:
Are you sure you want to download this stream? It will take 2.4GB to download and you currently have 14GB of space left
I've checking properties like countOfBytesReceived and countOfBytesExpectedToReceive but these wont give back correct values.
let headRequest = NSMutableURLRequest(URL: asset.streamURL)
headRequest.HTTPMethod = "HEAD"
let sizeTask = NSURLSession.sharedSession().dataTaskWithRequest(headRequest) { (data, response, error) in
print("Expected size is \(response?.expectedContentLength)")
}.resume()
prints a size of 2464, where at the end the size is 3GB.
During the download I logged above properties:
func URLSession(session: NSURLSession, assetDownloadTask: AVAssetDownloadTask, didLoadTimeRange timeRange: CMTimeRange, totalTimeRangesLoaded loadedTimeRanges: [NSValue], timeRangeExpectedToLoad: CMTimeRange) {
print("Downloaded \( convertFileSizeToMegabyte(Float(assetDownloadTask.countOfBytesReceived)))/\(convertFileSizeToMegabyte(Float(assetDownloadTask.countOfBytesExpectedToReceive))) MB")
}
But these stay at zero:
Downloaded 0.0/0.0 MB
HLS streams are actually a collection of files known as manifests and transport streams. Manifests usually contain a text listing of sub-manifests (each one corresponding to a different bitrate), and these sub-manifests contains a list of transport streams that contain the actual movie data.
In your code, when you download the HLS URL, you're actually downloading just the master manifest, and that's typically a few thousand bytes. If you want to copy the entire stream, you'll need to parse all the manifests, replicate the folder structure of the original stream, and grab the transport segments too (these are usually in 10-second segments, so there can be hundreds of these). You may need to rewrite URLs if the manifests are specified with absolute URLs as well.
To compute the size of each stream, you could multiply the bitrate (listed in the master manifest) by the duration of the stream; that might be a good enough estimate for download purposes.
A better answer here, since you're using the AVAssetDownloadTask in the context of offline FairPlay, is to implement the AVAssetDownloadDelegate. One of the methods in that protocol gives you the progress you're looking for:
URLSession:assetDownloadTask:didLoadTimeRange:totalTimeRangesLoaded:timeRangeExpectedToLoad:
Here's WWDC 2016 Session 504 showing this delegate in action.
There are a lot of details related to offline playback with FairPlay, so it's a good idea to go through that video very carefully.
I haven't worked with this API personally, but I am at least somewhat familiar with HTTP Live Streaming. With that knowledge, I think I know why you're not able to get the info you're looking for.
The HLS protocol is designed for handling live streaming as well as streaming of fixed-length assets. It does this by dicing up the media into in what are typically about ten-second chunks, IIRC, and listing the URLs for those chunks in a playlist file at a specific URL.
If the playlist does not change, then you can download the playlist, calculate the number of files, get the length of the first file, and multiply that by the number of files, and you'll get a crude approximation, which you can replace with an exact value when you start retrieving the last chunk.
However there is no guarantee that the playlist will not change. With HLS, the playlist can potentially change every ten seconds, by removing the oldest segments (or not) and adding new segments at the end. In this way, HLS supports streaming of live broadcasts that have no end at all. In that context, the notion of the download having a size is nonsensical.
To make matters worse, 2464 is probably the size of the playlist file, not the size of the first asset in it, which is to say that it tells you nothing unless that subclass's didReceiveResponse: method works, in which case you might be able to obtain the length of each segment by reading the Content-Length header as it fetches it. And even if it does work normally, you probably still can't obtain the number of segments from this API (and there's also no guarantee that all the segments will be precisely the same length, though they should be pretty close).
I suspect that to obtain the information you want, even for a non-live asset, you would probably have to fetch the playlist, parse it yourself, and perform a series of HEAD requests for each of the media segment URLs listed in it.
Fortunately, the HLS specification is a publicly available standard, so if you want to go down that path, there are RFCs you can read to learn about the structure of the playlist file. And AFAIK, the playlist itself isn't encrypted with any DRM or anything, so it should be possible to do so even though the actual decryption portion of the API isn't public (AFAIK).
This is my C#/Xamarin code to compute the final download size. It is most likely imperfect, especially with the new codecs supported with iOS11, but you should get the idea.
private static async Task<long> GetFullVideoBitrate(string manifestUrl)
{
string bandwidthPattern = "#EXT-X-STREAM-INF:.*(BANDWIDTH=(?<bitrate>\\d+)).*";
string videoPattern = "^" + bandwidthPattern + "(RESOLUTION=(?<width>\\d+)x(?<height>\\d+)).*CODECS=\".*avc1.*\".*$";
string audioPattern = "^(?!.*RESOLUTION)" + bandwidthPattern + "CODECS=\".*mp4a.*\".*$";
HttpClient manifestClient = new HttpClient();
Regex videoInfoRegex = new Regex(videoPattern, RegexOptions.Multiline);
Regex audioInfoRegex = new Regex(audioPattern, RegexOptions.Multiline);
string manifestData = await manifestClient.GetStringAsync(manifestUrl);
MatchCollection videoMatches = videoInfoRegex.Matches(manifestData);
MatchCollection audioMatches = audioInfoRegex.Matches(manifestData);
List<long> videoBitrates = new List<long>();
List<long> audioBitrates = new List<long>();
foreach (Match match in videoMatches)
{
long bitrate;
if (long.TryParse(match.Groups["bitrate"]
.Value,
out bitrate))
{
videoBitrates.Add(bitrate);
}
}
foreach (Match match in audioMatches)
{
long bitrate;
if (long.TryParse(match.Groups["bitrate"]
.Value,
out bitrate))
{
audioBitrates.Add(bitrate);
}
}
if (videoBitrates.Any() && audioBitrates.Any())
{
IEnumerable<long> availableBitrate = videoBitrates.Where(b => b >= Settings.VideoQuality.ToBitRate());
long videoBitrateSelected = availableBitrate.Any() ? availableBitrate.First() : videoBitrates.Max();
long totalAudioBitrate = audioBitrates.Sum();
return videoBitrateSelected + totalAudioBitrate;
}
return 0;
}
We have built a system where videos are stored in mongodb. The videos are each a couple of hundred megabytes in size. The system is built in python3 using mongoengine. The c extensions of pymongo and bson are installed.
The definition of the mongoengine documents is:
class VideoStore(Document, GeneralMixin):
video = EmbeddedDocumentListField(SingleFrame)
mutdat = DateTimeField()
_collection = 'VideoStore'
def gen_video(self):
for one_frame in self.video:
yield self._get_single_frame(one_frame)
def _get_single_frame(self, one_frame):
if one_frame.frame.tell() != 0:
one_frame.frame.seek(0)
return pickle.loads(one_frame.frame.read())
class SingleFrame(EmbeddedDocument):
frame = FileField()
Reading a video in Linux takes about 3 to 4 seconds. However running the same code in Windows takes 13 to 17 seconds.
Does anyone out there have any experience with this problem and any kind of solution?
I have thought of and tested (to no avail):
increasing the chunksize
reading the video as a single blob without using yield
Storing the file as a single blob (so no storing of separate frames)
Use Linux, Windows is poorly supported. The use of "infinite" virtual memory among other things causes issues with Windows variants. This thread elaborates further:
Why Mongodb performance better on Linux than on Windows?
I have to calculate a hash from various streams (StringIO, File, chunked http responses...), and the sources are pretty big (around 100MB - 1GB). For example, I have the following code
require 'digest'
sha = Digest::SHA256.new
stream = StringIO.new("test\nfoo\nbar\nhello world")
# this could also be a File.open('my_file.txt')
# or a chunked http response
while content = stream.read(2)
sha.update content
end
puts sha.to_s
This works so far, but I was wondering how the sha.update method works. Does it store a copy from the overall String in its instance, so that the whole content is hold in memory?
This could lead to some serious memory issues, when loading 1GB of data into RAM (and doing this on multiple processes on the same machine)
I'm writing custom firmware for a SparkFun Logomatic V2 that records binary data to a file on a 2GB micro-SD card. The data file size will range from 100 MB to 1 GB.
The format of the binary data is in flux as the board's firmware evolves (it will actually be dynamically reconfigurable at run-time). Rather than create and maintain a separate decoder/converter program for each version of firmware/configuration, I'd much rather make the data files self-converting to CSV format by starting the data file with a Bash script that is written to the data file before data recording starts.
I know how to create a Here Document, but I suspect Bash would be unable to quickly parse and convert a gigabyte of binary data, so I'd like to make the process run much faster by having the script first compile some C code (assume GCC is present and in the path), then run the resulting program, passing the binary data to stdin.
To make the problem more concrete, assume the firmware will create binary data consisting of 4 16-bit integer values: A timestamp (unsigned) followed by 3 accelerometer axes (signed). There is no separator between records (mainly because I'm saturating the SPI interface to the uSD card).
So, I think I need a script with TWO here documents: One for the C code (parameterized by expanded Bash variables), and another for the binary data. Here's where I am so far:
#! env bash
# Produced by firmware version 0.0.0.0.0.1 alpha
# Configuration for this data run:
header_string = "Time, X, Y, Z"
column_count = 4
# Create the converter executable
# Use "<<-" to permit code to be indented for readability.
# Allow variable expansion/substitution.
gcc -xc /tmp/convertit - <<-THE_C_CODE
#include <stdio.h>
int main (int argc, char **argv) {
// Write ${header_string} to stdout
while (1) {
// Read $(column_count} shorts from stdin
// Break if EOF
// Write $(column_count} comma-delimited values to stdout
}
// Close stdout
return 0;
}
THE_C_CODE
# Pass the binary data to the converter
# Hard-quote the Here tag to prevent subsequent expansion/substitution
/tmp/convertit >./$1.csv <<'THE_BINARY_DATA'
...
... hundreds of megabytes of semi-random data ...
...
THE_BINARY_DATA
rm /tmp/convertit
exit 0
Does that look about right? I don't yet have a real data file to test this with, but I wanted to verify the idea before going much further.
Will Bash complain if the closing lines are missing? This may happen if data capture terminates unexpectedly due to a shock knocking loose the battery or uSD card. Or if the firmware borks.
Is there a faster or better method I should consider? For example, I wonder if Bash will be too slow to copy the binary data as fast as the C program can consume it: Should the C program open the data file directly?
TIA,
-BobC
You may want to have a look at makeself. It allows you to change any .tar.gz archive into a self-extracting file which is platform independent (something like a shell script that contains a here document). This will allow you to easily distribute your data and decoder. It also allows you to configure a script contained within the archive to be run when the container script is run. This way you can use makeself for packaging and inside the archive you can put your data files and decoder written in C or bash or whatever language you find suitable.
While it is possible to decode binary data using shell tools (e.g. using od), it's very cumbersome and ineffective. I'd recommend using either a C program or perl which is also likely to be found on almost any machine (check this page).