I have two video streaming units capable of streaming live video inputs:
AXIS Q7424-R Video Encoder
EPIPHAN VGADVI Broadcaster 99460 -
I am using gstreamer to view these streams on client terminals running linux. I am interested in the h264, rtp multicast streams (which both units support).
I can stream the Epiphan video using the following gstreamer pipeline:
gst-launch-0.10 udpsrc multicast-group=ADDRESS port=PORT caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! ffdec_h254 ! autovideosink
However, this pipeline does not work for the Axis unit as I get the following error repeatedly:
ffmpeg:0:: non-existing PPS referenced
ffmpeg:0:: non-existing PPS 0 referenced
ffmpeg:0:: decode_slice_header error
ffmpeg:0:: no frame!
ffdec_h264: decoding error (len:-1, have_data: 0)
I have read that this error means that the ffmpeg decoder is missing the SPS/PPS information provided by a keyframe. The axis unit has a GOV parameter which is the interval at which i-frames are sent; it is set to 32.
Note that I can view both units' rtp streams in unicast with the following:
gst-launch-0.10 rtspsrc location=rtsp://ADDRESS:PORT/... latency=100 ! rtph264depay ! ffdec_h264 ! autovideosink
Since unicast works and the unicast and multicast pipelines are the same (except for the source), my guess is either:
My udpsrc caps are simply incorrect for the axis stream (and I don't really know where/how to verify it)
or, the axis multicast format/encoding is different and requires a modification to the pipeline (I find this unlikely since unicast is working and I don't understand why the encoding would change between unicast/multicast).
Any suggestions are appreciated as I am limited by my knowledge of gstreamer and media formats in terms of what to try next.
As stated in szatmary's comments, the axis hardware does not seem to stream the SPS/PPS information. I have contacted AXIS' support on this issue to which I received a response stating that my question is "outside of the scope of support staff".
The solution I have found was to include the "sprop-parameter-sets" in the receiving pipeline. This parameter seems to be unique per stream and can be obtained by either:
starting a unicast receiver with rtsp (example provided in question above) which will provide the set of caps that can be copied, or
accessing the .sdp file from the hardware, which is usually available through http (for example, http://<USERNAME:PASSWORD>#<ADDRESS:PORT>/axis-cgi/alwaysmulti.sdp?camera=1)
Note that accessing the sdp file is per stream (hence the camera=1), so if your hardware has multiple inputs then be sure to grab the right one.
Related
I need help for debugging a probabilistic issue. I built a gstreamer pipeline to stream NVENC encoded h264 bitstreams(video only) to browser. Browser seldom plays properly. In most cases only few frames are rendered then the picture gets frozen.
The NVENC settings follow "https://cloud.google.com/solutions/gpu-accelerated-streaming-using-webrtc" which are h264 high profile & low latency high quality & NVENC_INFINITE_GOPLENGTH(some settings have been tried, like rateControlMode/enableVFR/sliceMode/repeatSPSPPS/outputAUD but no help). At runtime, NVENC encodes real-time rendered opengl fbo texture to h264 bitstreams and push them into gstreamer via appsrc. Currently the texture size is 512x512 and fed at 10/20/30 fps.
I use gstreamer 1.18.2, the pipeline is defined as "appsrc name=nvenc_src do-timestamp=1 ! video/x-h264, stream-format=byte-stream, alignment=au ! rtph264pay aggregate-mode=zero-latency ! queue ! application/x-rtp,media=video,encoding-name=H264,payload=123 ! webrtcbin bundle-policy=max-compat name=backend_webrtc".
The gstreamer part codes follow the sendrecv example(replacing libsoup with websocketpp and removing the recv logics).
The application is built as MSVC 2019 32-bit. The browser decoder is NVDEC. Exe application and js codes run on the same PC(windwos 10, gtx1060, driver version 460.89). I've tried in Chrome(87.0.4280.88) and edge(87.0.664.66). I also tried running the js codes in android(chrome) and ios(safari) and get the same results.
It can be concluded that NVENC generates 'correct' h264 bitstreams. I dump the raw h264 bitstreams into file. The file plays properly in VLC. I also tried pushing the dumped h264 bitstreams into gstreamer. The frozen issue still happens.
After the picture is frozen, the playback never recovers. the browser's 'webrtc-internals' shows that bytes/headerBytes/packests_Received keep growing, while frameReceived/framesDecoded/framesDropped stay unchanged.
Since the bitwise same h264 frames behave differently at different runs, I guess rtp timestamps might cause the issue. I've tried setting appsrc's do-timestamp to 0 and manually set gstbuffer's PTS but it does not help.
Here are few things that you need to pay attention to:
Infinite GOP will not work - you must configure NVENC to send a key frame every 30 - 60 frames.
Of course SPS-PPS NALs must come before each key frame.
Prohibit B-frames: WebRTC doesn't support them because they increase latency.
Startup codes between NALs must be 3-bytes startup codes: WebRTC doesn't respect 2-bytes startup codes. We bumped into this issue before and had to manually correct the startup codes.
Thanks to user1390208's kind reminds, I use an h264 analyzer tool to check the dumpped bitstreams and find the evils.
Browser does support infinite GOP. But it needs keyframe & SPS-PPS for recovering from error. The need for resend arises with a high probability during the launch procedure. So a quick solution is resending keyframe & SPS-PPS after js detects fps is 0 and send a resend request to gstreamer via the webrtc data channel.
The reasons I failed to find the answer are two:
I didn't set encodePicFlags before calling nvEncEncodePicture. NVENC always generates infinite GOP whether gopLength & frameIntervalP are set to all I or I&P. There are many GOP related parameters and look confusing to me now. In my current codes, the only way to get the desried GOP control is by setting NV_ENC_PIC_PARAMS::encodePicFlags before calling nvEncEncodePicture. Note that I use NV_ENC_PRESET_LOW_LATENCY_HQ_GUID & NV_ENC_H264_PROFILE_HIGH_GUID which might cause infinite GOP is always generated(when encodePicFlags is not set)? NVENC reports no error when settings gopLength & frameIntervalP & repeatSPSPPS, so I thought the issue also happens when the GOP is all of I frames and SPS-PPS does not help.
Infinity GOP does not always cause the mentioned issue in browser during the launch procedure.
So before I check the h264 bitstreams with analyzer tool, it looks to me that even all-keyframe+SPS-PPS bitstreams have this probability issue.
BTW, NVENC generates 4-byte start code.
How do you stream a raw H.265 incoming video using Red5?
I've seen this example to stream flv file, and this for the client side, and for H.264 with or without ffmpeg.
Basically the question can be split into two:
How do you stream it from a .h265 file? If from .265 file is not possible, how do you do it from a file that contains H.265 video? Any example?
How do you stream it from an incoming RTP session? I can get the session UDP/RTP unpacked and decode into raw H.265 NAL packets. I'm assuming some conversion is needed, any libraries available for that purpose? Examples?
If I can get an answer to the above first split question, I know I can redirect the incoming stream to a named pipe. That can be used as an indirect solution to the second split question. Streaming from the incoming UDP session directly is preferred, though.
This is some preliminary idea, though surely not the best solution.
From a previous article "How to stream in h265 using gstreamer", there is a solution to include the x265enc element to gstreamer.
From an aws kinesis video examples page, a command line can be used to process rtsp/rtp input in h.265 and convert it to h.264 format:
gst-launch-1.0 rtspsrc location="rtsp://192.168.1.<x>:8554/h265video" ! \
decodebin ! x264enc ! <video-sink>
The <video-sink> should be something specific to Red5 server. Or if streaming in h.265 format which Red5 might or might not take:
gst-launch-1.0 rtspsrc location="rtsp://192.168.1.<x>:8554/h265video" \
short-header=TRUE ! rtph265depay ! h265parse ! video/x-h265, alignment=au ! \
<video-sink>
How can I use FFMPEG to add a delay to a stream being sent from a (v4l2) webcam to a media server?
The use case here is something like a security camera where I want to be able to stream video to a server when something is detected in the video. The easiest way to ensure the event of interest is captured on the video is to use FFMPEG to stream from the camera to a virtual loopback device with an added delay. That loopback device can then be used to initiate live streaming when an even of interest occurs.
In GStreamer, I would accomplish a delay of this sort with the queue element's min-threshold-time parameter. For example the following (much-simplified) example pipeline adds a 2 second delay to the output coming from a v4l2 webcam before displaying it:
gst-launch-1.0 v4l2src device=/dev/video1 ! queue max-size-buffers=0 max-size-time=0 max-size-bytes=0 min-threshold-time=2000000000 ! xvimagesink
How do I accomplish the same thing with FFMPEG? There are some technical challenges that prevent us from using GStreamer for this.
I have investigated the itsoffset option for this, but as far as I can tell it is only usable for already-recorded files, and it is not clear what a good alternative would be.
With a recent git build of ffmpeg, basic template is
ffmpeg -i input -vf tpad=start_duration=5 -af adelay=5000|5000 stream-out
The tpad filter will add 5 seconds of black at the start of the video stream, and the apad filter will add 5000 milliseconds of silence to the first two channels of the audio.
what I want to do is that send live camera stream which is encoded by h264 to gstreamer. I already have seen many example which send over network by using rtp and mpeg-ts. But problem is that all those examples assume that the input will be served by fixed file or live stream which is already transcoded in transport portocol like below.
client :
gst-launch-1.0 videotestsrc horizontal-speed=5 ! x264enc tune="zerolatency" threads=1 ! mpegtsmux ! tcpserversink host=192.168.0.211 port=8554
server : gst-launch-1.0 tcpclientsrc port=8554 host=192.168.0.211 ! tsdemux ! h264parse ! avdec_h264 ! xvimagesink
But, My camera offer the below interface (written in java, actually work on adnroid). The interface offer just live raw h264 blocks.
mReceivedVideoDataCallBack=newDJIReceivedVideoDataCallBack(){
#Override
public void onResult(byte[] videoBuffer, int size)
{
}
I can create tcp session to send those data block. But, how can i make those data which is not packed in transport protocol into format which is understable by gstreamer tcpclient?
Transcode the original stream in ts format in the camera side can be a solution. But i have no clue to do transcode from non-file and non-transport-format data. I have searched gstreamer and ffmpeg, But I could not derive a way to deal h264 block stream using the supported interface, unitl now.
Or, Are there any way to make gstreamer to directly accept those simple raw h264 block?
I think the best solution is to create your own element for your video source and then construct a pipeline using your element and mpegtsmux.
However, you can use appsrc + mpegtsmux and feed your appsrc through JNI with buffers you have from callback.
I'm working with FFMpeg for decoding Mjpeg streams.
Recently I've bumped into access violation exceptions from FFMpeg, after investigating, I found that due to network packet drops, I'm passing to the FFMpeg a frame that might have "gaps" in it.
The FFMpeg probably crash since it jumps to a marker payload which doesn't exist in the frame's memory.
Any idea where I can find a mjpeg structure validator?
Is there any way to configure FFMpeg to perform such validations by itself?
Thanks.
I would be inclined to use Gstreamer here instead of ffmpeg and set "max-errors" property in jpegdec plugin to -1.
gst-launch -v souphttpsrc location="http://[ip]:[port]/[dir]/xxx.cgi" do-timestamp=true is_live=true ! multipartdemux ! jpegdec max-errors=-1 ! ffmpegcolorspace ! autovideosink.
This takes care of the corrupt jpeg frames and continues the stream.
Didn't really found an answer to the question.
Apparently, ffmpeg doesn't handle corrupted frames very well.
Decided to try a different 3rd party decoder instead of ffmpeg. For now, at least for Jpeg, it works faster and much more robust.