I am trying to understand the MPEG streaming over network. I have couple of questions specific to MPEG-2 and MPEG-4 streaming.
1) As I understood MPEG-2 uses a Transport Stream (TS) which is encapsulated within RTP/UDP or directly in UDP/TCP packets for transmission. My question is which one is actually implemented by broadcast operators these days?
2) I read that broadcast operators are supporting MPEG-4 for spectral efficiency these days. Now, MPEG-4 transmission can be done in various ways. As per RFC3640 the Elementary Streams with Access Units can be transmitted over RTP; Or, encapsulated NAL unit in RTP packet in case of H.264 (RFC 3984); Or, MPEG-4 encoded data can also be transmitted using a Transport Stream (which is actually designed for MPEG-2). Now, in context of current days broadcasting which transport mechanism is used for MPEG-4?
The standard is called ATSC. Over the air is NOT the internet. There is no IP hence no UDP/TCP/RTP/etc. Its just an MPEG transport stream with FEC. In the case of mpeg-4 codec, specifically AVC is multiplexed within the transport stream.
https://en.wikipedia.org/wiki/Advanced_Television_Systems_Committee_standards
Related
I am using libvlc for RTSP h.264 bitstream decoding and display in PC. For the best experience (i.e. low latency), I uses the following options
:file-caching=0
:tcp-caching=0
:rtsp-caching=0
:network-caching=0
:clock-jitter=0
:avcodec-fast
With these parameters, the latency is acceptable in comparison to the open-source project "ONVIF device manager"(ODM) where FFMPEG is used for decoding.
When RTSP server deliver low frame-rate RTSP stream, (1 frame/second). These configuration will freeze after displaying a few frames (3-5 frames).
I have tried 2 different approaches
Disable synchronization by
:clock-synchro=0
This will enable the decoding process to go on, however, an accumulated period of time lagging could be observed.
Use network-cache
My experiments shows that
:network-caching=1200
Will make decoding go smoothly, however, the latency is over 1-2 second in comparison to ODM.
Is there a way to handle the low frame rate issue in libvlc without providing such big latency?
My application do encoding of captured frame from GDI or DXGI method. currently i am doing encoding with help x264 library.
AFAIK x264 is software based library, i want to do encoding with help of GPU, so it can save CPU cycles and hope speed also will be faster.
After searching, I found a H.264 Video Encoder MFT which is doing h264 encoding.
But couple of questions are answered for me.
1) is It faster than x264 encoding library?
2) can bitmap frame be encoded with help this MFT?
- i have seen only MFVideoFormat_I420, MFVideoFormat_IYUV, MFVideoFormat_NV12, MFVideoFormat_YUY2, MFVideoFormat_YV12 these formats are supported
3) is it hardware accelerated(mean it's using CPU or GPU)?
- Initially my understanding was it uses GPU but i get confused after reading this post MFT Encoder (h264) High CPU utilization.
4) can H.264 Video Encoder MFT be used stand alone without using sink writer, as i have to sent data on network?
5) is there any other alternative in windows?
It might be some questions are very silly, please feel free to edit.
Media Foundation H.264 Video Encoder is software encoder. From my [subjective] experience it slower than x264 and, perhaps more important, x264 offers wider range of settings, specifically when it comes to choose modes on the speed over quality end of the range. Either way, stock MS encoder is not hardware accelerated.
However, there might be other MFTs available (typically installed with respective hardware drivers) that do hardware accelerated H.264 encoding. You can discover them by enumerating MFTs, perhaps most popular is Intel Quick Sync Video (QSV) Encoder.
HardwareVideoEncoderTransform app does the enumeration and provides you with relevant details:
Typical input is NV12, some offer other input choices (such as e.g. 32-bit RGB). If you need other formats, you will have to pre-convert the input.
Hardware backed encoders CPU consumption is low, and their efficiency depends on the hardware implementation. Yes, you can use them standalone, entirely standalone or wrapped as DirectShow filter and included in normal DirectShow pipeline.
Alternate H.264 encoders are typically SDK based, or wrappers over those SDKs in DirectShow/MFT form factors because vendors package their implementation in well-known forms already familiar to multimedia developers.
I understand that if the hardware decoders are not present then we need to send pcm samples to alsa, but is it solely because of hardware limitation, or is there any requirement from ALSA as well ?
You need to use a sample format that is
known to ALSA, and
supported by the device.
I'm looking for the fastest way to encode a webcam stream that will be viewable in a html5 video tag. I'm using a Pandaboard: http://www.digikey.com/product-highlights/us/en/texas-instruments-pandaboard/686#tabs-2 for the hardware. Can use gstreamer, cvlc, ffmpeg. I'll be using it to drive a robot, so need the least amount of lag in the video stream. Quality doesn't have to be great and it doesn't need audio. Also, this is only for one client so bandwidth isn't an issue. The best solution so far is using ffmpeg with a mpjpeg gives me around 1 sec delay. Anything better?
I have been asked this many times so I will try and answer this a bit generically and not just for mjpeg. Getting very low delays in a system requires a bit of system engineering effort and also understanding of the components.
Some simple top level tweaks I can think of are:
Ensure the codec is configured for the lowest delay. Codecs will have (especially embedded system codecs) a low delay configuration. Enable it. If you are using H.264 it's most useful. Most people don't realize that by standard requirements H.264 decoders need to buffer frames before displaying it. This can be upto 16 for Qcif and upto 5 frames for 720p. That is a lot of delay in getting the first frame out. If you do not use H.264 still ensure you do not have B pictures enabled. This adds delay to getting the first picture out.
Since you are using mjpeg, I don't think this is applicable to you much.
Encoders will also have a rate control delay. (Called init delay or vbv buf size). Set it to the smallest value that gives you acceptable quality. That will also reduce the delay. Think of this as the bitstream buffer between encoder and decoder. If you are using x264 that would be the vbv buffer size.
Some simple other configurations: Use as few I pictures as possible (large intra period).
I pictures are huge and add to the delay to send over the network. This may not be very visible in systems where end to end delay is in the range of 1 second or more but when you are designing systems that need end to end delay of 100ms or less, this and several other aspects come into play. Also ensure you are using a low latency audio codec aac-lc (and not heaac).
In your case to get to lower latencies I would suggest moving away from mjpeg and use at least mpeg4 without B pictures (Simple profile) or best is H.264 baseline profile (x264 gives a zerolatency option). The simple reason you will get lower latency is that you will get lower bitrate post encoding to send the data out and you can go to full framerate. If you must stick to mjpeg you have close to what you can get without more advanced features support from the codec and system using the open source components as is.
Another aspect is the transmission of the content to the display unit. If you can use udp it will reduce latency quite a lot compared to tcp, though it can be lossy at times depending on network conditions. You have mentioned html5 video. I am curious as to how you are doing live streaming to a html5 video tag.
There are other aspects that can also be tweaked which I would put in the advanced category and requires the system engineer to try various things out
What is the network buffering in the OS? The OS also buffers data before sending it out for performance reasons. Tweak this to get a good balance between performance and speed.
Are you using CR or VBR encoding? While CBR is great for low jitter you can also use capped vbr if the codec provides it.
Can your decoder start decoding partial frames? So you don't have to worry about framing the data before providing it to the decoder. Just keep pushing the data to the decoder as soon as possible.
Can you do field encoding? Halves the time from frame encoding before getting the first picture out.
Can you do sliced encoding with callbacks whenever a slice is available to send over the network immediately?
In sub 100 ms latency systems that I have worked in all of the above are used. Some of the features may not be available in open source components but if you really need it and are enthusiastic you could go ahead and implement them.
EDIT:
I realize you cannot do a lot of the above for a ipad streaming solution and there are limitations because of hls also to the latency you can achieve. But I hope it will prove useful in other cases when you need any low latency system.
We had a similar problem, in our case it was necessary to time external events and sync them with the video stream. We tried several solutions but the one described here solved the problem and is extremely low latency:
Github Link
It uses gstreamer transcode to mjpeg which is then sent to a small python streaming server. This has the advantage that it uses the tag instead of so it can be viewed by most modern browsers, including the iPhone.
As you want the <video> tag, a simple solution is to use http-launch. That
had the lowest latency of all the solutions we tried so it might work for you. Be warned that ogg/theora will not work on Safari or IE so those wishing to target the Mac or Windows will have to modify the pipe to use MP4 or WebM.
Another solution that looks promising, gst-streaming-server. We simply couldn't find enough documentation to make it worth pursuing. I'd grateful if somebody could ask a stackoverflow question about how it should be used!
I am making a VOIP application for mobile platform. My question is what algorithms should be used to calculate whether the RTP package is "expired".
I transfer PCMU encoded audio wrapped into RTP packages via UDP.
As you know some of the datagrams are not delivered while others are delivered late and its pointless to play the audio in those packages.
Using the sequence number in the RTP header you can calculate the lost packages, but I want to know how to calculate when a packet is late.
I saw that there is something called jitter which basically measures the difference between the times for sending two consecutive packages and receiving them.
Can I use that? Or something else?
What your application considers 'expired' or 'too late' is really up to your application, but you should make sure you play out the audio evenly. So the measure for too late is the size of your playout buffer and the size of the buffer depends on the type of application. Bidirectional communication will need a smaller buffer than simple movie playback.