I'm working with vlcj, there i combined CaptureTest and StreamRtpDuplicate to achieve streaming from a capture device.
streaming from a capture device is working fine at the receiver end with 4-6 seconds delay,but local display got frozen at the first frame.
we are trying with manually entered option showing bellow,
mediaPlayer.playMedia(media, ":dshow-size=1280*720", ":dshow-fps=24", ":sout=#transcode{vcodec=h264,vb=800,acodec=mpga,ab=128,channels=2,samplerate=44100}:duplicate{dst=udp{mux=ts, dst=192.168.1.13:10000},dst=display}", ":sout-all", ":sout-keep") ;
where media = "dshow://"
system specifications:
Windows 10 64 bit
RAM : 4GB
Processor : i5-7400
Java version : 32 bit 1.8.0_161
JRE : 1.8.0_161-b12
Eclipse : oxygen 2
VLCj : 3.11.0
VLC : 2.1.5 Rincewind 32 bit
Capture Device : Logitech HD ProWebcam C920
Console part:
[185cf584] x264 encoder: profile High, level 3.1
[185cf584] x264 encoder: final ratefactor: 30.29
[swscaler # 22df4c00] deprecated pixel format used, make sure you did set range correctly
[185cf584] x264 encoder: using SAR=3/4
[185cf584] x264 encoder: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[185cf584] x264 encoder: profile High, level 3.1
[268690bc] main vout display error: Failed to set on top
[268690bc] main vout display error: Failed to resize display
[22ec6adc] avcodec decoder error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
[22ec6adc] avcodec decoder error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
[22ec6adc] avcodec decoder error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
.
.
.
.
when media file is streamed both local and receiver display is working fine
Please help me out to fixing the local display issue without disturbing receiver end.
Related
I would like to pass raw audio and video buffers from ffmpeg to gstreamer. For video a 1920x1080 25fps RGB output is what ffmpeg is producing.
What is the best method to pass this from ffmpeg to gstreamer on the same hardware.
The end goal is to not block either ffmpeg from outputting if gstreamer cannot take the frame and to not block gstreamer if no frames are available.
For this we have looked at sockets and tcp/udp plugins. However if we get any buffering issues the gstreamer pipline will block until the buffer is clear/full.
we will have multiple pairs of TX/RX running on same linux instance so stdI/O will not work.
Is there a current preferred method for this type of transfer?
I need help for debugging a probabilistic issue. I built a gstreamer pipeline to stream NVENC encoded h264 bitstreams(video only) to browser. Browser seldom plays properly. In most cases only few frames are rendered then the picture gets frozen.
The NVENC settings follow "https://cloud.google.com/solutions/gpu-accelerated-streaming-using-webrtc" which are h264 high profile & low latency high quality & NVENC_INFINITE_GOPLENGTH(some settings have been tried, like rateControlMode/enableVFR/sliceMode/repeatSPSPPS/outputAUD but no help). At runtime, NVENC encodes real-time rendered opengl fbo texture to h264 bitstreams and push them into gstreamer via appsrc. Currently the texture size is 512x512 and fed at 10/20/30 fps.
I use gstreamer 1.18.2, the pipeline is defined as "appsrc name=nvenc_src do-timestamp=1 ! video/x-h264, stream-format=byte-stream, alignment=au ! rtph264pay aggregate-mode=zero-latency ! queue ! application/x-rtp,media=video,encoding-name=H264,payload=123 ! webrtcbin bundle-policy=max-compat name=backend_webrtc".
The gstreamer part codes follow the sendrecv example(replacing libsoup with websocketpp and removing the recv logics).
The application is built as MSVC 2019 32-bit. The browser decoder is NVDEC. Exe application and js codes run on the same PC(windwos 10, gtx1060, driver version 460.89). I've tried in Chrome(87.0.4280.88) and edge(87.0.664.66). I also tried running the js codes in android(chrome) and ios(safari) and get the same results.
It can be concluded that NVENC generates 'correct' h264 bitstreams. I dump the raw h264 bitstreams into file. The file plays properly in VLC. I also tried pushing the dumped h264 bitstreams into gstreamer. The frozen issue still happens.
After the picture is frozen, the playback never recovers. the browser's 'webrtc-internals' shows that bytes/headerBytes/packests_Received keep growing, while frameReceived/framesDecoded/framesDropped stay unchanged.
Since the bitwise same h264 frames behave differently at different runs, I guess rtp timestamps might cause the issue. I've tried setting appsrc's do-timestamp to 0 and manually set gstbuffer's PTS but it does not help.
Here are few things that you need to pay attention to:
Infinite GOP will not work - you must configure NVENC to send a key frame every 30 - 60 frames.
Of course SPS-PPS NALs must come before each key frame.
Prohibit B-frames: WebRTC doesn't support them because they increase latency.
Startup codes between NALs must be 3-bytes startup codes: WebRTC doesn't respect 2-bytes startup codes. We bumped into this issue before and had to manually correct the startup codes.
Thanks to user1390208's kind reminds, I use an h264 analyzer tool to check the dumpped bitstreams and find the evils.
Browser does support infinite GOP. But it needs keyframe & SPS-PPS for recovering from error. The need for resend arises with a high probability during the launch procedure. So a quick solution is resending keyframe & SPS-PPS after js detects fps is 0 and send a resend request to gstreamer via the webrtc data channel.
The reasons I failed to find the answer are two:
I didn't set encodePicFlags before calling nvEncEncodePicture. NVENC always generates infinite GOP whether gopLength & frameIntervalP are set to all I or I&P. There are many GOP related parameters and look confusing to me now. In my current codes, the only way to get the desried GOP control is by setting NV_ENC_PIC_PARAMS::encodePicFlags before calling nvEncEncodePicture. Note that I use NV_ENC_PRESET_LOW_LATENCY_HQ_GUID & NV_ENC_H264_PROFILE_HIGH_GUID which might cause infinite GOP is always generated(when encodePicFlags is not set)? NVENC reports no error when settings gopLength & frameIntervalP & repeatSPSPPS, so I thought the issue also happens when the GOP is all of I frames and SPS-PPS does not help.
Infinity GOP does not always cause the mentioned issue in browser during the launch procedure.
So before I check the h264 bitstreams with analyzer tool, it looks to me that even all-keyframe+SPS-PPS bitstreams have this probability issue.
BTW, NVENC generates 4-byte start code.
I'm debugging an audio codec and need to test if the data is coming out of a codec properly. I have my code dumping the raw pcm stream into a file and now I want to try to play it in VLC or another OS X compatible player. I've heard that this is possible with VLC from the command line.
Short of writing some code to output your own header followed by your PCM data which will of course work - you could use Audacity to import the file as RAW data where you then specify :
Encoding : signed 16 bit PCM
Byte order : little endian (typically unless you're on powerpc or such)
Channels : 1 mono
start offset : 0
sample rate 44100 Hz (typical)
once imported Audacity will let you play the clip ... not as slick as some command line but easy to do none the less
this also works
ffplay -autoexit -f s16le myrawaudio.pcm
above assumes your raw audio is using bit depth of 16 bits which is typical of CD quality audio
I am having issues on ffmpeg encoded video files audio tracks. My encoded video contains ID for audio track as extracted from mediainfo:
I use CODEC_ID_MP3 in guess_format "mov" container for quicktime
Audio
ID : 2
Format : MPEG Audio
Format version : Version 2
Format profile : Layer 3
Codec ID : .mp3
Duration : 2s 916ms
Bit rate mode : Constant
Bit rate : 128 Kbps
Channel(s) : 1 channel
Sampling rate : 16.0 KHz
Compression mode : Lossy
Stream size : 45.3 KiB (2%)
Language : English
This is not recognized on a vanilla codecless installation of windows 7, only played by k-lite codec libmad
I have noticed, that another test file contains similar mp3 track and is played by media player OK:
Audio
ID : 2
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 3
Mode : Joint stereo
Mode extension : MS Stereo
Codec ID : 6B
Duration : 1mn 9s
Bit rate mode : Constant
Bit rate : 320 Kbps
Channel(s) : 2 channels
Sampling rate : 44.1 KHz
Compression mode : Lossy
Stream size : 2.67 MiB (38%)
Writing library : LAME3.98
The difference I noticed is in Format version number and Codec ID, which is Version 2 from ffmpeg output - I couldnt locate this version setting in ffmpeg source files, so my question is - is there a way to influence this format version identificator and set the codec id as in above playable video?
I had to hack ffmpeg sources to change the codec ID to the one I needed
Version 1 only supports sample frequencies of 32 kHz, 44.1 kHz, and 48 kHz. For 16 kHz you need version 2, or you could resample to 32 kHz (-ar 32k) if you need version 1.
If necessary you can override the codec id with -atag id (and -strict unofficial if you are doing something contrary to the standard and want ffmpeg to not be bothered by that).
I am working on a project that needs to decode the h264 video using dxva2.0. I write the code according to the documentation http://msdn.microsoft.com/en-us/library/windows/desktop/aa965245%28v=vs.85%29.aspx. That is to say, I create an interface of IDirectXVideoDecoder, then I call the dxva api "BeginFrame", "Execute" and "EndFrame". Then the problem comes out. when I execute my program on a Intel Core i5 CPU(the GPU is the Intel HD graphics inside the CPU), everything is OK. But when I execute it on a intel ATOM processor(with the intel GMA3000 series graphics hardware), I can't get the right result: some video frames are decoded correctly, whereas others are totally in mess.
The data I use is sent from another computer, and the data can be directly filled to the buffers of dxva. In h264, the buffers are DXVA2_PictureParameter, DXVA2_Bitstream, DXVA2_InverseQuantization and DXVA2_SliceControl. So it is not necessary to use the ffmpeg or ffdshow(and the ffdshow is gpl, I can't use it).
The "dxva checker" software tells me the guid of intel core i5 is "ModeH264_VLD_NoFGT_ClearVideo" and that of intel atom is "ModeH264_VLD_NoFGT". I want to know the differences between the two guids.
Could it be possible to use "ModeH264_VLD_NoFGT" on a intel video card to decode a video?