very low latency streaminig with ffmpeg using a webcam - ffmpeg

I'm trying to configure ffmpeg to do a real-time video streaming using a webcam. The ffmpeg encoder command I use is as follows.
ffmpeg -f v4l2 -input_format yuyv422 -s 640x480 -i /dev/video0 -c:v libx264 -profile:v baseline -trellis 0 -subq 1 -level 32 -preset superfast -tune zerolatency -me_method epzs -crf 30 -threads 0 -bufsize 1 -refs 4 -coder 0 -b_strategy 0 -bf 0 -sc_threshold 0 -x264-params vbv-maxrate=2000:slice-max-size=1500:keyint=30:min-keyint=10: -pix_fmt yuv420p -an -f mpegts udp://192.168.1.8:5001
The ffplay command used to display the video feed is,
ffplay -analyzeduration 1 -fflags -nobuffer -i udp://192.168.1.8:5001
However, I'm experiencing a latency of 0.5 - 1.0s latency in the video stream. Is there a way to reduce this to a number less than 100ms. Also, when I replace the v4l2 camera capture with a screen capture using x11grab, the stream is almost real-time and I experience no noticeable delays. Moreover, changing the encoder from x264 to mpeg2 had no effect on the latency. In addition, the statistics from the ffmpeg shows that the encoder is performing at a 30fps rate, which I believe indicates that the encoding is real-time. This leaves me with only one reason for the experienced delay.
Is there a significant delay in buffers when using v4l2 during video capturing in a webcam?
I don't think the transmission delay is in effect in this case as I see no latencies when screen capture is used under the same conditions.
Can this latency be further reduced?. Can someone think of a different encoder configuration to be used instead of the one that I've been using?

I had also many problems in setting up a low latency video streaming system between an odroid spc and windows pc. Finally i found settings resulting in approx 500ms to max. 1s latency.
Setup: ffserver on odroid xu4 with ubuntu 18.04, connected via wifi dongle to network. Windows 10 PC in same wifi network streaming from odroid.
I run the following ffserver config (/etc/ffserver.conf) on my odroid
HTTPPort 8090
HTTPBindAddress 0.0.0.0
MaxClients 1000
MaxBandwidth 10000
<Feed feed1.ffm>
File /tmp/feed1.ffm
FileMaxSize 50M
ACL allow 127.0.0.1
ACL allow localhost
</Feed>
<Stream test1.asf>
Format asf
Feed feed1.ffm
VideoFrameRate 30
VideoSize 640x480
VideoBitRate 600
#VideoBufferSize 400
VideoQMin 1
VideoQMax 20
NoAudio
ACL allow 127.0.0.1
ACL allow localhost
ACL allow 192.168.0.0 192.168.255.255
</Stream>
<Stream stat.html>
Format status
ACL allow 127.0.0.1
ACL allow localhost
</Stream>
and start the camera stream on the odroid with
ffserver -f /etc/ffserver.conf & ffmpeg -f v4l2 -s 640x480 -r 15 -i /dev/video0 -vcodec libx265 -threads 2 -tune zerolatency http://localhost:8090/feed1.ffm
On my Windows PC I tried several settings to get low latency. With VLC-Player I could not manage anything below 8 to 10 seconds.
With the following ffplay command I got about 500ms latency:
ffplay -fflags -nobuffer -probesize 32 -i mmsh://ubuntu1804:8090/test1.asf
so, -sync ext and -analyzeduration 1 did not help in reducing the latency.
The "stream production" on the odroid also runs with the same low latency when using libx264 instead of libx265 and removing the -thread 2 flag. But increasing the framerate to 30 or even increasing the resolution leads to significant delays.

i used the same send instruction and i tried this with the ffplay and it worked for me:
ffplay -analyzeduration 1 -fflags -nobuffer -probesize 32 -sync ext -i rtmp://localhost/live/STREAM_NAME

Related

Possible for ffmpeg to downsample over time, serve RTSP?

I'd like to take an RSTP webcam, downsample the video to a lower rate (say one frame every 5 seconds) and serve the result as an RTSP stream.
Is is possible to configure ffmpeg (or libffmpeg) to do such a thing?
Yes, all we have to do is adding -r 0.2 argument, and re-encode the video.
It is also recommended to add -tune zerolatency or -g 1 for making sure every frame is a key frame (required in case video latency is relevant).
Example:
Receiving RTSP stream from localhost, and streaming at 0.2fps (to localhost with different port):
ffmpeg -rtsp_flags listen -rtsp_transport tcp -stimeout 1000000 -i rtsp://127.0.0.1:10000/live.stream -r 0.2 -vcodec libx264 -tune zerolatency -pix_fmt yuv420p -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:20000/live.stream
Testing:
For testing I simulated the RTSP camera with FFmpeg (streaming synthetic video at 25fps).
The RTSP stream is captured by another FFmpeg process that reduces the rate to 0.2fps.
The 0.2fps video is captured and displayed using FFprobe.
The test is implemented as a batch file:
::Play the video for testing
start ffplay -rtsp_flags listen -rtsp_transport tcp -flags low_delay -vf setpts=0 -listen_timeout 1000000 rtsp://127.0.0.1:20000/live.stream
::Wait 5 seconds
ping 127.0.0.1 -n 5 > nul
::Capture the RTSP camera at 25fps, convert to 0.2fps (with re-encoding)
start ffmpeg -rtsp_flags listen -rtsp_transport tcp -stimeout 1000000 -i rtsp://127.0.0.1:10000/live.stream -r 0.2 -vcodec libx264 -tune zerolatency -pix_fmt yuv420p -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:20000/live.stream
::Wait 5 seconds
ping 127.0.0.1 -n 5 > nul
::Simulate an RTSP camera at 25fps
ffmpeg -re -f lavfi -i testsrc=size=192x108:rate=25 -vcodec libx264 -pix_fmt yuv420p -g 30 -rtsp_transport tcp -f rtsp -muxdelay 0.1 rtsp://127.0.0.1:10000/live.stream
It starts awkward and gets stable after few frames.
(We mat use select filter for solving it).
Sample frames:

FFmpeg latency is over 3 seconds (must be subsecond)

I'm working on a robot (raspberry pi 4 based) that is accessible from anywhere. My robot is currently at a 3-second latency. I also use OvenMediaEngine (RTMP to WebRTC) to transmit my stream to the client (on a website).Here is my command:
raspivid -n -t 0 -w 1280 -h 720 -fps 25 -b 3500000 -g 50 -fl -o - | ffmpeg -thread_queue_size 1024 -i - -itsoffset 6 -f alsa -channels 1 -thread_queue_size 1024 -i hw:2 -preset ultrafast -tune zerolatency -vcodec libx264 -r 25 -b:v 512k -s 1280x720 -acodec aac -ac 2 -ab 32k -ar 44100 -f flv rtmp://xxxxxxxx:1935/app/stream
Does anyone know why it won't stream at subsecond latency?
Thanks in advance!
I am not exactly sure where you are incurring latency, but it usually happens either during transport or encoding.
If possible I would see if you avoid re-encoding to H264. You are going to pay a penalty of decoding (or just parsing?) and then encoding.
I would also see if you can ingest into OME with something other then RTMP. WebRTC and RTSP both will give you better latency.

ffmpeg to kurento rtp stream choppy video issue

I use ffmpeg to make an rtp stream to kurento media server and then send it to browsers via webrtc.
ffmpeg (h264 RTP) -> Kurento -> (h264 WebRTC) Browser
I'm capturing virtual xorg display.
This is my ffmpeg command:
ffmpeg -y -v info -fflags +genpts -f x11grab -draw_mouse 0 -r 25 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -an -c:v libx264 -preset veryfast -crf 25 -g 50 -pix_fmt yuv420p -maxrate 2976k -bufsize 5952k -ssrc 112233 -payload_type 103 -tune zerolatency -f rtp rtp://172.16.1.115:40258
This is my fake sdp offer used in negotiation with kurento RtpEndpoint
v=0
o=- 0 0 IN IP4 127.0.0.1
s=Stream
c=IN IP4 127.0.0.1
t=0 0
m=video 9 RTP/AVP 103
a=rtpmap:103 H264/90000
a=fmtp:103 packetization-mode=1
a=sendonly
a=direction:active
a=ssrc:112233 cname:user#example.com
Here is the problem:
Some I-frames are choppy on bottom half slice of the frame while others have no problem.
It is sometimes corrected when another I-frame has arrived, but mostly choppy.
When choppiness happened, Kms log says:
kmsutils kmsutils.c:483:gap_detection_probe:kmsagnosticbin2-108:sink Stream gap detected, timestamp: 0:51:22.574766908, duration: 0:00:00.000008237
Normal stream (No choppy at all)
Choppy stream
When it is corrected with an I-Frame (Sometimes it happens)
I have no clue about what could lead the issue.
Things that I tried to solve the problem.
Adding ?pkt_size=1000 (1100,900,1200. Since the mtu in kurento is 1200 default)
Changing -crf to different values in between 18-35
Changing -preset between medium and ultrafast
Changing framerate
Changing gop length (When i lower the gop length -More I-Frames- choppiness become shorter in duration but more frequent)
When I disable sliced-threads, there is no issue with bottom side but whole screen freezes in same scenario
Any help would be appreciated. Thanks.

What steps are needed to stream RTSP from FFmpeg?

What steps are needed to stream RTSP from FFmpeg?
Streaming UDP is not a problem, but as I want to stream to mobile devices which can natively read RTSP streams, I couldn't find any setup which tells what exactly is needed. Do I need an RTSP streaming server like LIVE555 or can I use FFmpeg only?
My Command:
ffmpeg -i space.mp4 -vcodec libx264 -tune zerolatency -crf 18 -f rtsp -muxdelay 0.1 rtsp://192.168.1.200:1234
I get an Input/Output error.
Do I need a SDP description to use RTSP?
And if yes where do I have to put it?
You can use FFserver to stream a video using RTSP.
Just change console syntax to something like this:
ffmpeg -i space.mp4 -vcodec libx264 -tune zerolatency -crf 18 http://localhost:1234/feed1.ffm
Create a ffserver.config file (sample) where you declare HTTPPort, RTSPPort and SDP stream. Your config file could look like this (some important stuff might be missing):
HTTPPort 1234
RTSPPort 1235
<Feed feed1.ffm>
File /tmp/feed1.ffm
FileMaxSize 2M
ACL allow 127.0.0.1
</Feed>
<Stream test1.sdp>
Feed feed1.ffm
Format rtp
Noaudio
VideoCodec libx264
AVOptionVideo flags +global_header
AVOptionVideo me_range 16
AVOptionVideo qdiff 4
AVOptionVideo qmin 10
AVOptionVideo qmax 51
ACL allow 192.168.0.0 192.168.255.255
</Stream>
With such setup you can watch the stream with i.e. VLC by typing:
rtsp://192.168.0.xxx:1235/test1.sdp
Here is the FFserver documentation.
FWIW, I was able to setup a local RTSP server for testing purposes using simple-rtsp-server and ffmpeg following these steps:
Create a configuration file for the RTSP server called rtsp-simple-server.yml with this single line:
protocols: [tcp]
Start the RTSP server as a Docker container:
$ docker run --rm -it -v $PWD/rtsp-simple-server.yml:/rtsp-simple-server.yml -p 8554:8554 aler9/rtsp-simple-server
Use ffmpeg to stream a video file (looping forever) to the server:
$ ffmpeg -re -stream_loop -1 -i test.mp4 -f rtsp -rtsp_transport tcp rtsp://localhost:8554/live.stream
Once you have that running you can use ffplay to view the stream:
$ ffplay -rtsp_transport tcp rtsp://localhost:8554/live.stream
Note that simple-rtsp-server can also handle UDP streams (i.s.o. TCP) but that's tricky running the server as a Docker container.
Another streaming command I've had good results with is piping the ffmpeg output to vlc to create a stream. If you don't have these installed, you can add them:
sudo apt install vlc ffmpeg
In the example I use an mpeg transport stream (ts) over http, instead of rtsp. I've tried both, but the http ts stream seems to work glitch-free on my playback devices.
I'm using a video capture HDMI>USB device that sets itself up on the video4linux2 driver as input. Piping through vlc must be CPU-friendly, because my old dual-core Pentium CPU is able to do the real-time encoding with no dropped frames. I've also had audio-sync issues with some of the other methods, where this method always has perfect audio-sync.
You will have to adjust the command for your device or file. If you're using a file as input, you won't need all that v4l2 and alsa stuff. Here's the ffmpeg|vlc command:
ffmpeg -thread_queue_size 1024 -f video4linux2 -input_format mjpeg -i /dev/video0 -r 30 -f alsa -ac 1 -thread_queue_size 1024 -i hw:1,0 -acodec aac -vcodec libx264 -preset ultrafast -crf 18 -s hd720 -vf format=yuv420p -profile:v main -threads 0 -f mpegts -|vlc -I dummy - --sout='#std{access=http,mux=ts,dst=:8554}'
For example, lets say your server PC IP is 192.168.0.10, then the stream can be played by this command:
ffplay http://192.168.0.10:8554
#or
vlc http://192.168.0.10:8554
UPDATE:
Here is a command to use VLC for rtsp, instead of using the rtsp-simple-server:
ffmpeg -thread_queue_size 1024 -f video4linux2 -input_format mjpeg -video_size 1280x720 -r 30 -i /dev/video0 -f alsa -thread_queue_size 1024 -i plughw:CARD=MS2109,DEV=0 -acodec mp2 -vcodec libx264 -preset ultrafast -crf 20 -s hd720 -vf format=yuv420p -profile:v main -f mpegts -|vlc -I dummy - --sout='#rtp{sdp=rtsp://:8554/} --sout-all --sout-keep'
If your PC ip is 192.168.0.10, then the rtsp stream is played by this command:
vlc rtsp://192.168.0.10:8554/
An alternative that I used instead of FFServer was Red5 Pro. On Ubuntu, I used this line:
ffmpeg -f pulse -i default -f video4linux2 -thread_queue_size 64 -framerate 25 -video_size 640x480 -i /dev/video0 -pix_fmt yuv420p -bsf:v h264_mp4toannexb -profile:v baseline -level:v 3.2 -c:v libx264 -x264-params keyint=120:scenecut=0 -c:a aac -b:a 128k -ar 44100 -f rtsp -muxdelay 0.1 rtsp://localhost:8554/live/paul

FFMPEG real-time buffer full frame dropped

I'm using FFMPEG on windows with direct show.
I'm streaming RTMP (command below) and i need very low latency.
Once run I get the following errors: [dshow # 024ce800] real-time buffer 204% full! frame dropped!
ffmpeg -threads 6 -f dshow -i video=UScreenCapture -s 1920x1080 -an -vco
dec libx264 -x264opts keyint=25:min-keyint=20 -b:v 1024k -preset ultrafast -tune zerolatency -crf 22 -r 10 -pix_fmt yuv420p -f flv rtmp://server...
Do you have an idea how to handle this kind of error?
Thanks
Ronen
That message means "dshow got an incoming packet, but you haven't finished sending the previous packet yet" so in reality, this should be contributing to as low a latency possible. If your goal is to avoid dropping packets then increase rtbufsize. With rtmp hopefully there will be some improvements soon so it has better throughput.

Resources