My application do encoding of captured frame from GDI or DXGI method. currently i am doing encoding with help x264 library.
AFAIK x264 is software based library, i want to do encoding with help of GPU, so it can save CPU cycles and hope speed also will be faster.
After searching, I found a H.264 Video Encoder MFT which is doing h264 encoding.
But couple of questions are answered for me.
1) is It faster than x264 encoding library?
2) can bitmap frame be encoded with help this MFT?
- i have seen only MFVideoFormat_I420, MFVideoFormat_IYUV, MFVideoFormat_NV12, MFVideoFormat_YUY2, MFVideoFormat_YV12 these formats are supported
3) is it hardware accelerated(mean it's using CPU or GPU)?
- Initially my understanding was it uses GPU but i get confused after reading this post MFT Encoder (h264) High CPU utilization.
4) can H.264 Video Encoder MFT be used stand alone without using sink writer, as i have to sent data on network?
5) is there any other alternative in windows?
It might be some questions are very silly, please feel free to edit.
Media Foundation H.264 Video Encoder is software encoder. From my [subjective] experience it slower than x264 and, perhaps more important, x264 offers wider range of settings, specifically when it comes to choose modes on the speed over quality end of the range. Either way, stock MS encoder is not hardware accelerated.
However, there might be other MFTs available (typically installed with respective hardware drivers) that do hardware accelerated H.264 encoding. You can discover them by enumerating MFTs, perhaps most popular is Intel Quick Sync Video (QSV) Encoder.
HardwareVideoEncoderTransform app does the enumeration and provides you with relevant details:
Typical input is NV12, some offer other input choices (such as e.g. 32-bit RGB). If you need other formats, you will have to pre-convert the input.
Hardware backed encoders CPU consumption is low, and their efficiency depends on the hardware implementation. Yes, you can use them standalone, entirely standalone or wrapped as DirectShow filter and included in normal DirectShow pipeline.
Alternate H.264 encoders are typically SDK based, or wrappers over those SDKs in DirectShow/MFT form factors because vendors package their implementation in well-known forms already familiar to multimedia developers.
Related
I'm developing a VoD application as a white label product that runs in a SaaS context using K8s. To enable streaming, I take the input video and re-convert it into HLS segments in multiple version and codecs to reach maximum compatibility.
Yesterday I started implementing AV1 as codec, as it will in near future detach h264 as it's more efficient with the same level of compatibility across all the available browsers.
That was the point where things started to get strange, as I want to have this codec instead of h264 ^^.
If you take a look at the following doc pages from ffmpeg: https://trac.ffmpeg.org/wiki/Encode/AV1
You will notice that there are 3 main encoders available to handle encoding to av1. These are: libaom, SVT-AV1 and rav1e. No matter which one of these I try, the performance is slow, even slower than with HEVC. Recently I came along a news article about Netflix and that they are upgrading their library to AV1. If I take a look at the numbers of media elements Netflix offers, the amount is just huge, and I really don't understand how they did it. From what I know, SVT-AV1 is developed by Netflix in cooperation with Intel, So I assume they somehow rely on hardware encoding using an Intel CPU extension.
Does somebody maybe know more and how they did it? I really can't imagine that they just do CPU only encoding. A movie would take days to get encoded.
Thanks in advance
Encoding quality and quality differs heavily between all encoders. SVT-AV1 is the fastest but looks like garbage. For real-time encoding you should probably use GPU's. Intels GPU's don't really put out great quality AV1 encodes though, Nvidia's H265 is basically the same quality.
With Nvidia and AMD soon getting AV1 encoding hardware support (currently drivers are a bit lacking but it's already possible on Nvidia). AMD GPU's coming out for it soon.
I am working on a project which captures screen and encodes it. I can already capture screen using desktop duplication API (Win8+). Using the API I can get ID3D11Texture2D textures and transfer them from GPU to CPU and then use libx264 to encode them.
However, pulling textures from GPU to CPU can be a bottle neck which potentially can decrease the fps. Also libx264 takes up CPU cycles (depending on quality) to encode frames. I am looking for encoding ID3D11Texture2D textures in GPU itself instead of using CPU for encoding as an optimization.
I have already checked the documentation and some sample codes but I have had no success. I would appreciate if someone could point me to some resource that does exactly what I want reliably.
Video encoders, hardware and software, might be available in different form factors. Windows itself offers extensible APIs with choice of encoders, and then additional encoders might be available as libraries and SDKs. You are already using one of such libraries (x264). Hardware encoders are typically vendor-specific and depend on available hardware, which is involved directly in the process of encoding. If you are interested in solution for specific hardware, it might make sense to check for respective SDK.
Otherwise, typical generic interface for hardware backed video encoding in Windows is Media Foundation Transform (MFT). Microsoft provides stock software only H.264 video encoder which is unlikely to give any advantage over x264, except the fact that it shares MFT interface with other options. Video hardware drivers, however, would often install additional MFTs for the hardware available, which add more MFTs backed by hardware implementation. Examples of such are:
Intel® Quick Sync Video H.264 Encoder MFT
NVIDIA H.264 Encoder MFT
AMDh264Encoder
Offered by different vendors, they offer similar functionality and your using these MFTs to encode H.264 is a good way to take advantage of hardware video encoding with a wide range of hardware.
See also:
Registering and Enumerating MFTs
Basic MFT Processing Model
You have to check if sharing texture between GPU encoder and DirectX is possible.
I know it's possible to share texture between Nvidia Decoder and DirectX, because i've done it successfully. Nvidia has some interop capacity, so first, look if you can share texture to do all things in GPU.
With Nvidia you can do this : Nvidia Decoding->DirectX Display in GPU.
Check if DirectX Display->Nvidia Enconding is possible (knowing nvidia offers Interop)
For Intel and ATI, i don't know if they provide interop with DirectX.
The main thing to know is to check if you can interop your DirectX texture with GPU encoder texture.
I would like to do a performance comparison between Ffmpeg and Intel Media SDK in transcoding.
I have to write a new application that will do the following.
Receive frames from MJPEG, MPEG4 and H264 cameras.
Transcode the frames. Output will be h264. Here I have to either use Ffmpeg or Intel Media SDK.
Multicast transcoded frames as RTSP streaming.
I have noticed that both these libs are CPU intensive. Is there any settings in Ffmpeg can reduce the CPU usage?
Thanks in advance,
As with all media encoding, you will trade speed for quality. x264 (ffmpeg) will produce higher quality (or smaller files at the same quality) but will use more CPU. Intel Media SDK should use very little CPU, but the quality will be a bit less. It accomplishes this by using specialized hardware on the CPU, if your CPU supports it.
So, what is best? It depends on want you want to optimize for CPU, Power usage, Quality, or file size.
I'm looking for the fastest way to encode a webcam stream that will be viewable in a html5 video tag. I'm using a Pandaboard: http://www.digikey.com/product-highlights/us/en/texas-instruments-pandaboard/686#tabs-2 for the hardware. Can use gstreamer, cvlc, ffmpeg. I'll be using it to drive a robot, so need the least amount of lag in the video stream. Quality doesn't have to be great and it doesn't need audio. Also, this is only for one client so bandwidth isn't an issue. The best solution so far is using ffmpeg with a mpjpeg gives me around 1 sec delay. Anything better?
I have been asked this many times so I will try and answer this a bit generically and not just for mjpeg. Getting very low delays in a system requires a bit of system engineering effort and also understanding of the components.
Some simple top level tweaks I can think of are:
Ensure the codec is configured for the lowest delay. Codecs will have (especially embedded system codecs) a low delay configuration. Enable it. If you are using H.264 it's most useful. Most people don't realize that by standard requirements H.264 decoders need to buffer frames before displaying it. This can be upto 16 for Qcif and upto 5 frames for 720p. That is a lot of delay in getting the first frame out. If you do not use H.264 still ensure you do not have B pictures enabled. This adds delay to getting the first picture out.
Since you are using mjpeg, I don't think this is applicable to you much.
Encoders will also have a rate control delay. (Called init delay or vbv buf size). Set it to the smallest value that gives you acceptable quality. That will also reduce the delay. Think of this as the bitstream buffer between encoder and decoder. If you are using x264 that would be the vbv buffer size.
Some simple other configurations: Use as few I pictures as possible (large intra period).
I pictures are huge and add to the delay to send over the network. This may not be very visible in systems where end to end delay is in the range of 1 second or more but when you are designing systems that need end to end delay of 100ms or less, this and several other aspects come into play. Also ensure you are using a low latency audio codec aac-lc (and not heaac).
In your case to get to lower latencies I would suggest moving away from mjpeg and use at least mpeg4 without B pictures (Simple profile) or best is H.264 baseline profile (x264 gives a zerolatency option). The simple reason you will get lower latency is that you will get lower bitrate post encoding to send the data out and you can go to full framerate. If you must stick to mjpeg you have close to what you can get without more advanced features support from the codec and system using the open source components as is.
Another aspect is the transmission of the content to the display unit. If you can use udp it will reduce latency quite a lot compared to tcp, though it can be lossy at times depending on network conditions. You have mentioned html5 video. I am curious as to how you are doing live streaming to a html5 video tag.
There are other aspects that can also be tweaked which I would put in the advanced category and requires the system engineer to try various things out
What is the network buffering in the OS? The OS also buffers data before sending it out for performance reasons. Tweak this to get a good balance between performance and speed.
Are you using CR or VBR encoding? While CBR is great for low jitter you can also use capped vbr if the codec provides it.
Can your decoder start decoding partial frames? So you don't have to worry about framing the data before providing it to the decoder. Just keep pushing the data to the decoder as soon as possible.
Can you do field encoding? Halves the time from frame encoding before getting the first picture out.
Can you do sliced encoding with callbacks whenever a slice is available to send over the network immediately?
In sub 100 ms latency systems that I have worked in all of the above are used. Some of the features may not be available in open source components but if you really need it and are enthusiastic you could go ahead and implement them.
EDIT:
I realize you cannot do a lot of the above for a ipad streaming solution and there are limitations because of hls also to the latency you can achieve. But I hope it will prove useful in other cases when you need any low latency system.
We had a similar problem, in our case it was necessary to time external events and sync them with the video stream. We tried several solutions but the one described here solved the problem and is extremely low latency:
Github Link
It uses gstreamer transcode to mjpeg which is then sent to a small python streaming server. This has the advantage that it uses the tag instead of so it can be viewed by most modern browsers, including the iPhone.
As you want the <video> tag, a simple solution is to use http-launch. That
had the lowest latency of all the solutions we tried so it might work for you. Be warned that ogg/theora will not work on Safari or IE so those wishing to target the Mac or Windows will have to modify the pipe to use MP4 or WebM.
Another solution that looks promising, gst-streaming-server. We simply couldn't find enough documentation to make it worth pursuing. I'd grateful if somebody could ask a stackoverflow question about how it should be used!
I have a potential job which will require me to do some video encoding with FFMPEG and x264. I'll have a series of files which I'll need to encode once, then I'll be able to bring down the instances. Since I'm not really sure of the resource utilization of x264 and FFMPEG, what kind of instances should I get? I'm thinking either a
High-CPU Extra Large Instance
7 GB of memory
20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: c1.xlarge
or, alternatively a
Cluster GPU Quadruple Extra Large Instance
22 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
2 x NVIDIA Tesla “Fermi” M2050 GPUs
1690 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
API name: cg1.4xlarge
What should I use? Does x264/FFMPEG perform better with faster/more CPUs or does it really pound the GPU more? In any case, it seems that the Cluster GPU seems to be the higher performance instance. What should I prefer?
Ffmpeg recently added support for VAAPI and VDPAU, but this allows it to use the GPU only for decoding of H.264 video. For encoding, it uses the CPU.
The short answer to CPU/GPU is, it depends on how you use ffmpeg to do the H264 encoding.
Also, you are confusing the H264 and x264. H264 is the video codec standard and x264 is one implementation of H264 standard. x264 is so popular, so sometimes it has become synonymous and confused with H264. The reason that I point that out is x264 is a software-based implementation of H264, which means it will only use the CPU cores for all the processes. There will be no GPU usage in your use case when you use x264 for video encoding.
That being said, maybe what you are trying to ask is whether to go for
hardware-based implementation of H264 (which uses the GPU), or
software-based implementation of H264 (which uses the CPU)
There are several implementations available for each available. Ffmpeg already has a nice page on this. If you are planning to use the Nvidia GPU instances, then you would need to compile FFmpeg with NVENC support to get the hardware implementation. Using GPUs/CPUs to efficiently do all your transcoding process is an art itself.
So in short, x264 will not use GPU. If you want to use GPU, you need to use hardware implementations of the encoders. Which implementation is better largely depends upon your use case and what you care about (quality, cost, turnaround time, etc.)
My background/ Disclaimer: I work as a Senior Engineer at Bitmovin. We solve this "cluster/resource" allocation engineering problem, among many many many other problems, to extract the best possible video quality out of a given bitrate. And in the end, we offer APIs where you can just simply plug them into your workflow. The views expressed here are my own.
In the present, Amazon EC2 offers (some) GPU accelerated instances using modern NVIDIA GPUs, meaning that you can take advantage of NVENC on them.
You are probably better off using a service like zencoder.com, they have an excellent API and the quality you will get out of it will most probably be better than hours of fiddling with Ffmpeg parameters optimisation.