I'm programming video capturing app and need to have 2 input sources (USB cams) to record from at the same time.
When I record only the raw footage simultaneously without compression at is working quite well (Low CPU load, no video lags), but when the compression is turned on the CPU is very high and the footage is lagging.
How to solve it? Or how to tune-up the settings so that it can be accomplished?
Note: the Raw streams are to big and thus cannot be used, otherwise I would not bother with compression at all and just leave it as it is.
The AVFoundation framework in its current configuration is setup to provide HW acceleration only for one source at time. For multiple accelerated sources one need to go deeper to VideoToolbox framework and even deeper.
Related
I'm developing a VoD application as a white label product that runs in a SaaS context using K8s. To enable streaming, I take the input video and re-convert it into HLS segments in multiple version and codecs to reach maximum compatibility.
Yesterday I started implementing AV1 as codec, as it will in near future detach h264 as it's more efficient with the same level of compatibility across all the available browsers.
That was the point where things started to get strange, as I want to have this codec instead of h264 ^^.
If you take a look at the following doc pages from ffmpeg: https://trac.ffmpeg.org/wiki/Encode/AV1
You will notice that there are 3 main encoders available to handle encoding to av1. These are: libaom, SVT-AV1 and rav1e. No matter which one of these I try, the performance is slow, even slower than with HEVC. Recently I came along a news article about Netflix and that they are upgrading their library to AV1. If I take a look at the numbers of media elements Netflix offers, the amount is just huge, and I really don't understand how they did it. From what I know, SVT-AV1 is developed by Netflix in cooperation with Intel, So I assume they somehow rely on hardware encoding using an Intel CPU extension.
Does somebody maybe know more and how they did it? I really can't imagine that they just do CPU only encoding. A movie would take days to get encoded.
Thanks in advance
Encoding quality and quality differs heavily between all encoders. SVT-AV1 is the fastest but looks like garbage. For real-time encoding you should probably use GPU's. Intels GPU's don't really put out great quality AV1 encodes though, Nvidia's H265 is basically the same quality.
With Nvidia and AMD soon getting AV1 encoding hardware support (currently drivers are a bit lacking but it's already possible on Nvidia). AMD GPU's coming out for it soon.
I compiled ffmpeg and h264 libraries for android using NDK.
I am recording videos using the muxing.c example from the ffmpeg library. Everything works correct (still haven't worked on the audio) but the camera is dropping frames and it takes around 100ms to save each frame, which is unacceptable.
I have also tried making a queue and saving them into another thread (let's call it B) but at the end I need to wait for around 120 seconds because the background thread (B)is still recording the frames.
Is there a workaround for this issue, besides reducing the video size? Ideally I would like to save the frames in real time, at least reduce the saving time. Is it just that Android is incapable of doing this? .
First of all, check if you can be better served by the hardware encoder (via MediaRecorder or MediaCodec in Java, or using OpenMax from native code).
If for some reason you must encode in software, and your device is multicore, you can gain a lot by compiling x264 to use sliced multithreading. Let me cite my post of 2 years ago:
We are using x264 directly (no ffmpeg code involved), and with ultafast/zerolatency preset we get 30 FPS for VGA on Samsung Note10 (http://www.gsmarena.com/samsung_galaxy_note_10_1_n8000-4573.php) with Quad-core 1.4 GHz Cortex-A9 Exynos 4412 CPU, which is on the paper weaker than Droid's Quad-core 1.5 GHz Krait Qualcomm MDM615m/APQ8064 (http://www.gsmarena.com/htc_droid_dna-5113.php).
Note that x264 build scripts do not enable pthreads for Android (because NDK does not include libpthread.a), bit you can build the library with multithread support (very nice for a Quad-core CPU) if you simply create a dummy libpthread.a, see https://mailman.videolan.org/pipermail/x264-devel/2013-March/009941.html.
Note that encoder setup is only one part of the problem. If you work with the deprecated camera API, you should use preallocated buffers and a background thread for camera callbacks, as I explained elsewhere.
For a project due to perform soon enough I happen to have a problem. Task is to play 4k PRORES 422 files according to a sequence that is written on a file (XML) while listening on a OSC port for CUE signals and some feedback to the operator. Player also can smooth the speed +- 15% and has a general fader, and sends back to OSC controller few data to update the performer.
The playback is now unpredictably un-smooth And I don't know why.
I also tend to think is not a problem of hardware: Machine used is a mac pro 2014 with 10.9 (https://www.apple.com/mac-pro/specs/) with 64mb Ram, all data on SDD and a hell of a graphic card. The un-smoothness in playback is rather unpredictable, random frame drops in different places. I tried to use external time on the player and is a bit better but still not satisfactory. I am going to package it without the editor in a app, but on preliminary test is not that faster.
I wander also what is the best way to examine the code for leaks...
Playback of files in quicktime player uses 20% of cpu and in quartz composer +90%
I am stuck on this issue, having done all the obvious things I think, and would like at least to understand how to profile the performance of the patch to find what is wrong and were.
Suggestions are welcome and thanks for help!
If its not interactive you could try rendering it in quicktime player.
I have a machine with 2x3 3ghz dual-core xeon and 4x10krpm scsi 320 disks in raid0.
The capture card is an osprey 560 64 bit pci card.
Operating system is currently Windows Server 2003.
The video-stream that I can open with VLC using direct show is rather nice quality.
However, trying to save this video-stream without loss of quality has proven quite difficult,
using the h264 codec I am able to achieve a satisfying quality, however, all 4 cores jump to 100% load after a few second and then it start dropping frames, the machine is not powerful enough for realtime encoding. I've not been able to achieve satisfying mpeg1 or 4 quality, no matter which bitrate I set..
Thing is, the disks in this machine are pretty fast even by todays standard, and they are bored.. I don't care about disk-usage, I want quality.
I have searched in vain for a way to pump that beautiful videostream that I see in VLC onto the disk for later encoding, I reckon the disks would be fast enough, or maybe something which would apply a light compression, enough that the disks can keep up, but not so much as to loose visible quality.
I have tried FFMPEG as it seems capable of streaming a yuv4 stream down to the disk, but ofcause FFMPEG is unable to open the dshow device ( same error as this guy Ffmpeg streaming from capturing device Osprey 450e fails )
Please recommend a capable and (preferably) software which can do this.
I finally found it out, it is deceptively simple:
Just uncheck the "transcode" option in the window where you can select an encoding preset.
2 gigabytes per minutes is a low price to pay for finally getting those precious memories off of old videotapes in the quality they deserve.
I'm looking for the fastest way to encode a webcam stream that will be viewable in a html5 video tag. I'm using a Pandaboard: http://www.digikey.com/product-highlights/us/en/texas-instruments-pandaboard/686#tabs-2 for the hardware. Can use gstreamer, cvlc, ffmpeg. I'll be using it to drive a robot, so need the least amount of lag in the video stream. Quality doesn't have to be great and it doesn't need audio. Also, this is only for one client so bandwidth isn't an issue. The best solution so far is using ffmpeg with a mpjpeg gives me around 1 sec delay. Anything better?
I have been asked this many times so I will try and answer this a bit generically and not just for mjpeg. Getting very low delays in a system requires a bit of system engineering effort and also understanding of the components.
Some simple top level tweaks I can think of are:
Ensure the codec is configured for the lowest delay. Codecs will have (especially embedded system codecs) a low delay configuration. Enable it. If you are using H.264 it's most useful. Most people don't realize that by standard requirements H.264 decoders need to buffer frames before displaying it. This can be upto 16 for Qcif and upto 5 frames for 720p. That is a lot of delay in getting the first frame out. If you do not use H.264 still ensure you do not have B pictures enabled. This adds delay to getting the first picture out.
Since you are using mjpeg, I don't think this is applicable to you much.
Encoders will also have a rate control delay. (Called init delay or vbv buf size). Set it to the smallest value that gives you acceptable quality. That will also reduce the delay. Think of this as the bitstream buffer between encoder and decoder. If you are using x264 that would be the vbv buffer size.
Some simple other configurations: Use as few I pictures as possible (large intra period).
I pictures are huge and add to the delay to send over the network. This may not be very visible in systems where end to end delay is in the range of 1 second or more but when you are designing systems that need end to end delay of 100ms or less, this and several other aspects come into play. Also ensure you are using a low latency audio codec aac-lc (and not heaac).
In your case to get to lower latencies I would suggest moving away from mjpeg and use at least mpeg4 without B pictures (Simple profile) or best is H.264 baseline profile (x264 gives a zerolatency option). The simple reason you will get lower latency is that you will get lower bitrate post encoding to send the data out and you can go to full framerate. If you must stick to mjpeg you have close to what you can get without more advanced features support from the codec and system using the open source components as is.
Another aspect is the transmission of the content to the display unit. If you can use udp it will reduce latency quite a lot compared to tcp, though it can be lossy at times depending on network conditions. You have mentioned html5 video. I am curious as to how you are doing live streaming to a html5 video tag.
There are other aspects that can also be tweaked which I would put in the advanced category and requires the system engineer to try various things out
What is the network buffering in the OS? The OS also buffers data before sending it out for performance reasons. Tweak this to get a good balance between performance and speed.
Are you using CR or VBR encoding? While CBR is great for low jitter you can also use capped vbr if the codec provides it.
Can your decoder start decoding partial frames? So you don't have to worry about framing the data before providing it to the decoder. Just keep pushing the data to the decoder as soon as possible.
Can you do field encoding? Halves the time from frame encoding before getting the first picture out.
Can you do sliced encoding with callbacks whenever a slice is available to send over the network immediately?
In sub 100 ms latency systems that I have worked in all of the above are used. Some of the features may not be available in open source components but if you really need it and are enthusiastic you could go ahead and implement them.
EDIT:
I realize you cannot do a lot of the above for a ipad streaming solution and there are limitations because of hls also to the latency you can achieve. But I hope it will prove useful in other cases when you need any low latency system.
We had a similar problem, in our case it was necessary to time external events and sync them with the video stream. We tried several solutions but the one described here solved the problem and is extremely low latency:
Github Link
It uses gstreamer transcode to mjpeg which is then sent to a small python streaming server. This has the advantage that it uses the tag instead of so it can be viewed by most modern browsers, including the iPhone.
As you want the <video> tag, a simple solution is to use http-launch. That
had the lowest latency of all the solutions we tried so it might work for you. Be warned that ogg/theora will not work on Safari or IE so those wishing to target the Mac or Windows will have to modify the pipe to use MP4 or WebM.
Another solution that looks promising, gst-streaming-server. We simply couldn't find enough documentation to make it worth pursuing. I'd grateful if somebody could ask a stackoverflow question about how it should be used!