Parsing FFMpeg AVPackets into h264 nal units - ffmpeg

I am using FFMpeg To decode live video and stream it using Live555.i am able to decode video and getting the output AVPackets.
1. Convert the BGR Image to YUV422P format using FFMpeg's SWScale
// initilize a BGR To RGB converter using FFMpeg
ctx = sws_getContext(codecContext->width, codecContext->height, AV_PIX_FMT_BGR24, codecContext->width, codecContext->height, AV_PIX_FMT_YUV422P, SWS_BICUBIC, 0, 0, 0);
tempFrame = av_frame_alloc();
int num_bytes = avpicture_get_size(PIX_FMT_BGR24, codecContext->width, codecContext->height);
uint8_t* frame2_buffer = (uint8_t*)av_malloc(num_bytes*sizeof(uint8_t));
avpicture_fill((AVPicture*)tempFrame, frame2_buffer, PIX_FMT_BGR24, codecContext->width, codecContext->height);
// inside the loop of where frames are being encoded where rawFrame is a BGR image
tempFrame->data[0] = reinterpret_cast<uint8_t*>(rawFrame->_data);
sws_scale(ctx, tempFrame->data, tempFrame->linesize, 0, frame->height, frame->data, frame->linesize);
For decoding each Frame
ret = avcodec_encode_video2(codecContext, &packet, frame, &got_output);
if(ret < 0)
{
fprintf(stderr, "Error in encoding frame\n");
exit(1);
}
if(got_output)
{
//printf("Received frame! pushing to queue\n");
OutputFrame *outFrame = new OutputFrame();
outFrame->_data = packet.buf->data;
outFrame->_bufferSize = packet.buf->size;
outputQueue.push_back(outFrame);
}
Till here it works fine. i am able to write these frames to file and play it using VLC. after this i have to pass the output frame to Live555.i think AVPackets i am getting here doesn't need to be a single H264 Nal unit which is required by Live555.
How to break a AVPacket into Nal units which can be passed to Live555?

H264VideoStreamDiscreateFramer expect data without the start code '\x00\x00\x00\x01'.
It is needed to remove the 4 first bytes either in your LiveDeviceSource or inserting a FramedFilter to do this job.
Perhaps you can tried to use an H264VideoStreamFramer, like the testH264VideoStreamer test program.
If it could help, you can find one of my tries with live555 implementing an RTSP server feed from V4L2 capture https://github.com/mpromonet/h264_v4l2_rtspserver

Related

RTMP live stream directly from NVENC encoder

I am trying to create a live RTMP stream containing the animation generated with NVIDIA OptiX. The stream is to be received by nginx + rtmp module and broadcasted in MPEG-DASH format. Full chain up to dash.js player is working if the video is first saved to .flv file and then I send it with ffmpeg without any reformatting using command:
ffmpeg -re -i my_video.flv -c:v copy -f flv rtmp://x.x.x.x:1935/dash/test
But I want to stream directly from the code. And with this I am failng... Nginx logs an error "dash: invalid avcc received (2: No such file or directory)". Then it seems to receive the stream correctly (segments are rolling, dash manifest is there), however the stream is not possible to play in the browser.
I can see only one difference in the manifest between direct stream and stream from file. Codecs attribute of the representation in the direct stream is missed: codecs="avcc1.000000" instead of "avc1.640028" which I get when streaming from file.
My code opens the stream:
av_register_all();
AVOutputFormat* fmt = av_guess_format("flv",
file_name, nullptr);
fmt->video_codec = AV_CODEC_ID_H264;
AVFormatContext* _oc;
avformat_alloc_output_context2(&_oc, fmt, nullptr, "rtmp://x.x.x.x:1935/dash/test");
AVStream* _vs = avformat_new_stream(_oc, nullptr);
_vs->id = 0;
_vs->time_base = AVRational { 1, 25 };
_vs->avg_frame_rate = AVRational{ 25, 1 };
AVCodecParameters *vpar = _vs->codecpar;
vpar->codec_id = fmt->video_codec;
vpar->codec_type = AVMEDIA_TYPE_VIDEO;
vpar->format = AV_PIX_FMT_YUV420P;
vpar->profile = FF_PROFILE_H264_HIGH;
vpar->level = _level;
vpar->width = _width;
vpar->height = _height;
vpar->bit_rate = _avg_bitrate;
avio_open(&_oc->pb, _oc->filename, AVIO_FLAG_WRITE);
avformat_write_header(_oc, nullptr);
Width, height, bitrate, level and profile I get from NVENC encoder settings. I also do the error checking, ommited here. Then I have a loop writing each encoded packets, with IDR frames etc all prepared on the fly with NVENC. The loop body is:
auto & pkt_data = _packets[i];
AVPacket pkt = { 0 };
av_init_packet(&pkt);
pkt.pts = av_rescale_q(_n_frames++, AVRational{ 1, 25 }, _vs->time_base);
pkt.duration = av_rescale_q(1, AVRational{ 1, 25 }, _vs->time_base);
pkt.dts = pkt.pts;
pkt.stream_index = _vs->index;
pkt.data = pkt_data.data();
pkt.size = (int)pkt_data.size();
if (!memcmp(pkt_data.data(), "\x00\x00\x00\x01\x67", 5))
{
pkt.flags |= AV_PKT_FLAG_KEY;
}
av_write_frame(_oc, &pkt);
Obviously ffmpeg is writing avcc code somewhere... I have no clue where to add this code so the RTMP server can recognize it. Or I am missing something else?
Any hint greatly appreciated, folks!
Thanks to Gyan's comment I was able to solve the issue. Following the AV_CODEC_FLAG_GLOBAL_HEADER flag in the wrapper one can see how the global header is added, which was missing in my case. You can use directly the NVENC API function nvEncGetSequenceParams, but since I am anyway using SDK, it is a bit cleaner.
So I had to attach the header to AVCodecParameters::extradata:
std::vector<uint8_t> payload;
_encoder->GetSequenceParams(payload);
vpar->extradata_size = payload.size();
vpar->extradata = (uint8_t*)av_mallocz(payload.size() + AV_INPUT_BUFFER_PADDING_SIZE);
memcpy(vpar->extradata, payload.data(), payload.size());
_encoder is my instance of NvEncoder from SDK.
The wrapper is doing the same thing, however using deprecated struct AVCodecContext.

FFmpeg avcodec_decode_video2 decode RTSP H264 HD-video packet to video picture with error

I used FFmpeg library version 4.0 to have simple C++ program, in witch is a thread to receive RTSP H264 video data from IP-camera and display it in program window.
Code of this thread is follow:
DWORD WINAPI GrabbProcess(LPVOID lpParam)
// Grabbing thread
{
DWORD i;
int ret = 0, nPacket=0;
FILE *pktFile;
// Open video file
pFormatCtx = avformat_alloc_context();
if(avformat_open_input(&pFormatCtx, nameVideoStream, NULL, NULL)!=0)
fGrabb=-1; // Couldn't open file
else
// Retrieve stream information
if(avformat_find_stream_info(pFormatCtx, NULL)<0)
fGrabb=-2; // Couldn't find stream information
else
{
// Find the first video stream
videoStream=-1;
for(i=0; i<pFormatCtx->nb_streams; i++)
if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO)
{
videoStream=i;
break;
}
if(videoStream==-1)
fGrabb=-3; // Didn't find a video stream
else
{
// Get a pointer to the codec context for the video stream
pCodecCtxOrig=pFormatCtx->streams[videoStream]->codec;
// Find the decoder for the video stream
pCodec=avcodec_find_decoder(pCodecCtxOrig->codec_id);
if(pCodec==NULL)
fGrabb=-4; // Codec not found
else
{
// Copy context
pCodecCtx = avcodec_alloc_context3(pCodec);
if(avcodec_copy_context(pCodecCtx, pCodecCtxOrig) != 0)
fGrabb=-5; // Error copying codec context
else
{
// Open codec
if(avcodec_open2(pCodecCtx, pCodec, NULL)<0)
fGrabb=-6; // Could not open codec
else
// Allocate video frame for input
pFrame=av_frame_alloc();
// Determine required buffer size and allocate buffer
numBytes=avpicture_get_size(pCodecCtx->pix_fmt, pCodecCtx->width,
pCodecCtx->height);
buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
// Assign appropriate parts of buffer to image planes in pFrame
// Note that pFrame is an AVFrame, but AVFrame is a superset
// of AVPicture
avpicture_fill((AVPicture *)pFrame, buffer, pCodecCtx->pix_fmt,
pCodecCtx->width, pCodecCtx->height);
// Allocate video frame for display
pFrameRGB=av_frame_alloc();
// Determine required buffer size and allocate buffer
numBytes=avpicture_get_size(AV_PIX_FMT_RGB24, pCodecCtx->width,
pCodecCtx->height);
bufferRGB=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
// Assign appropriate parts of buffer to image planes in pFrameRGB
// Note that pFrameRGB is an AVFrame, but AVFrame is a superset
// of AVPicture
avpicture_fill((AVPicture *)pFrameRGB, bufferRGB, AV_PIX_FMT_RGB24,
pCodecCtx->width, pCodecCtx->height);
// initialize SWS context for software scaling to FMT_RGB24
sws_ctx_to_RGB = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
AV_PIX_FMT_RGB24,
SWS_BILINEAR,
NULL,
NULL,
NULL);
// Allocate video frame (grayscale YUV420P) for processing
pFrameYUV=av_frame_alloc();
// Determine required buffer size and allocate buffer
numBytes=avpicture_get_size(AV_PIX_FMT_YUV420P, pCodecCtx->width,
pCodecCtx->height);
bufferYUV=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
// Assign appropriate parts of buffer to image planes in pFrameYUV
// Note that pFrameYUV is an AVFrame, but AVFrame is a superset
// of AVPicture
avpicture_fill((AVPicture *)pFrameYUV, bufferYUV, AV_PIX_FMT_YUV420P,
pCodecCtx->width, pCodecCtx->height);
// initialize SWS context for software scaling to FMT_YUV420P
sws_ctx_to_YUV = sws_getContext(pCodecCtx->width,
pCodecCtx->height,
pCodecCtx->pix_fmt,
pCodecCtx->width,
pCodecCtx->height,
AV_PIX_FMT_YUV420P,
SWS_BILINEAR,
NULL,
NULL,
NULL);
RealBsqHdr.biWidth = pCodecCtx->width;
RealBsqHdr.biHeight = -pCodecCtx->height;
}
}
}
}
while ((fGrabb==1)||(fGrabb==100))
{
// Grabb a frame
if (av_read_frame(pFormatCtx, &packet) >= 0)
{
// Is this a packet from the video stream?
if(packet.stream_index==videoStream)
{
// Decode video frame
int len = avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
nPacket++;
// Did we get a video frame?
if(frameFinished)
{
// Convert the image from its native format to YUV
sws_scale(sws_ctx_to_YUV, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameYUV->data, pFrameYUV->linesize);
// Convert the image from its native format to RGB
sws_scale(sws_ctx_to_RGB, (uint8_t const * const *)pFrame->data,
pFrame->linesize, 0, pCodecCtx->height,
pFrameRGB->data, pFrameRGB->linesize);
HDC hdc=GetDC(hWndM);
SetDIBitsToDevice(hdc, 0, 0, pCodecCtx->width, pCodecCtx->height,
0, 0, 0, pCodecCtx->height,pFrameRGB->data[0], (LPBITMAPINFO)&RealBsqHdr, DIB_RGB_COLORS);
ReleaseDC(hWndM,hdc);
av_frame_unref(pFrame);
}
}
// Free the packet that was allocated by av_read_frame
av_free_packet(&packet);
}
}
// Free the org frame
av_frame_free(&pFrame);
// Free the RGB frame
av_frame_free(&pFrameRGB);
// Free the YUV frame
av_frame_free(&pFrameYUV);
// Close the codec
avcodec_close(pCodecCtx);
avcodec_close(pCodecCtxOrig);
// Close the video file
avformat_close_input(&pFormatCtx);
avformat_free_context(pFormatCtx);
if (fGrabb==1)
sprintf(tmpstr,"Grabbing Completed %d frames", nCntTotal);
else if (fGrabb==2)
sprintf(tmpstr,"User break on %d frames", nCntTotal);
else if (fGrabb==3)
sprintf(tmpstr,"Can't Grabb at frame %d", nCntTotal);
else if (fGrabb==-1)
sprintf(tmpstr,"Couldn't open file");
else if (fGrabb==-2)
sprintf(tmpstr,"Couldn't find stream information");
else if (fGrabb==-3)
sprintf(tmpstr,"Didn't find a video stream");
else if (fGrabb==-4)
sprintf(tmpstr,"Codec not found");
else if (fGrabb==-5)
sprintf(tmpstr,"Error copying codec context");
else if (fGrabb==-6)
sprintf(tmpstr,"Could not open codec");
i=(UINT) fGrabb;
fGrabb=0;
SetWindowText(hWndM,tmpstr);
ExitThread(i);
return 0;
}
// End Grabbing thread
When program receive RTSP H264 video data with resolution 704x576 then decoded video pictures are OK. When receive RTSP H264 HD-video data with resolution 1280x720 it look like that first video picture is decoded OK and then video pictures are decoded but always with some error.
Please help me to fix this problem!
Here is problems brief :
I have an IP camera model HI3518E_50H10L_S39 (product of China).
Camera can provide H264 video stream both at resolution 704x576 (with RTSP URI "rtsp://192.168.1.18:554/user=admin_password=tlJwpbo6_channel=1_stream=1.sdp?real_stream") or 1280x720 (with RTSP URI "rtsp://192.168.1.18:554/user=admin_password=tlJwpbo6_channel=1_stream=0.sdp?real_stream").
Using FFplay utility I can access and display them with good picture quality.
For testing of grabbing from this camera, I have a simple (above mentioned) program in VC-2005. In "Grabbing thread" program use FFmpeg library version 4.0 for opening camera RTSP stream, retrieve stream information, find the first video stream... and prepare some variables.
Center of this thread is loop: Grab a frame (function av_read_frame) - Decode it if it's video (function avcodec_decode_video2) - Convert to RGB format (function sws_scale) - Display to program window (GDI function SetDIBitsToDevice).
When proram run with camera RTSP stream at resolution 704x576, I have good video picture. Here is a sample:
704x576 sample
When program run with camera RTSP stream at resolution 1280x720, first video picture is good:
First good at res.1280x720
but then not good:
not good at res.1280x720
Its seem to be my FFmpeg function call to avcodec_decode_video2 can't fully decode certain packet for some reasons.

How to configure AVStream to write 29.97FPS files using FFmpeg

I'm trying to write mkv file using ffmpeg to encode in FFV1 and FLAC in NTSC format, but the frame rate shown in VLC and media info are not correct.
Here is how I create and configure the output format context:
AVOutputFormat *outputFormat = av_guess_format("matroska", NULL, NULL);
//Allocate an AVFormatContext for an output format.
int err = avformat_alloc_output_context2(&_formatContext, outputFormat, NULL, filename);
//Specify the codec of the outputFormat
_formatContext->oformat->video_codec = _videoCodecContext->codec_id;
//Create AVStream
AVStream *videoStream = avformat_new_stream(_formatContext, NULL);
//FrameDuration.value : 1001, FrameDuration.timescale : 30000
videoStream->time_base = (AVRational){ (int)_frameDuration.value, (int)_frameDuration.timescale }; //1001 30000
//Copy video stream parameters to the muxer
err = avcodec_parameters_from_context(videoStream->codecpar, _videoCodecContext);
//Open file for writing
err = avio_open(&_formatContext->pb, filename, AVIO_FLAG_WRITE);
if (err >= 0) {
//Write header
err = avformat_write_header(_formatContext, &options);
}
Before writing the packet, I use this to convert PTS to the stream time_base
// Rescale output packet timestamp values from codec to stream timebase
av_packet_rescale_ts(inAVPacket, *inTimeStamp, [outputStream stream]->time_base);
The thing is that the avformat_write_header method is changing the stream time_base from 30000/1001 to 1/1000, so PTS loose precision. In VLC inspector, the frame rate shown is 1000 fps and in MediaInfo 30.033 fps.
The file is playing correctly and the video/audio sync is OK.
Is there something to do to specify the file frame rate somewhere else ?
Or a work around to avoid changing the time_base when calling avformat_write_header ?
Setting the avg_frame_rate fixes the issue...
videoStream->avg_frame_rate = _videoCodecContext->framerate;

Is it possible to decode MPEG4 frames without delay with ffmpeg?

I use ffmpeg's MPEG4 decoder. The decoder has CODEC_CAP_DELAY capability among others. It means the decoder will give me decoded frames with latency of 1 frame.
I have a set of MPEG4 (I- & P- )frames from AVI file and feed ffmpeg decoder with these frames. For the very first I-frame decoder gives me nothing, but decodes the frames successfully. I can force the decoder to get the decoded frame with the second call of avcodec_decode_video2 and providing nulls (flush it), but if I do so for each frame I get artifacts for the first group of pictures (e.g. second decoded P-frame is of gray color).
If I do not force ffmpeg decoder to give me decoded frame right now, then it works flawlessly and without artifacts.
Question: But is it possible to get decoded frame without giving the decoder next frame and without artifacts?
Small example of how decoding is implemented for each frame:
// decode
int got_frame = 0;
int err = 0;
int tries = 5;
do
{
err = avcodec_decode_video2(m_CodecContext, m_Frame, &got_frame, &m_Packet);
/* some codecs, such as MPEG, transmit the I and P frame with a
latency of one frame. You must do the following to have a
chance to get the last frame of the video */
m_Packet.data = NULL;
m_Packet.size = 0;
--tries;
}
while (err >= 0 && got_frame == 0 && tries > 0);
But as I said that gave me artifacts for the first gop.
Use the "-flags +low_delay" option (or in code, set AVCodecContext.flags |= CODEC_FLAG_LOW_DELAY).
I tested several options and "-flags low_delay" and "-probesize 32" is more important than others. bellow code worked for me.
AVDictionary* avDic = nullptr;
av_dict_set(&avDic, "flags", "low_delay", 0);
av_dict_set(&avDic, "probesize", "32", 0);
const int errorCode = avformat_open_input(&pFormatCtx, mUrl.c_str(), nullptr, &avDic);

How can i mux H264 stream into MP4 file via libavformat

I want to realize an application that firstly decode a multi-media file(such as test.mp4 file, video codec id is H264), get a video stream and an audio stream, then make some different in the audio stream, at last encode the video stream(use libx264) and audio stream into a result file(result.mp4). To promote the efficiency, i omitted the decode and encode of video stream, i get the video packet via function "av_read_frame", then output it directly into the result file via function "av_write_frame". But there is no picture in the output file, and the size of output file is fairly small.
I tracked the ffmpeg code and found that in the function "av_write_frame->mov_write_packet->ff_mov_write_packet", it will call function "ff_avc_parse_nal_units" to obtain the size of nal unit, but the return value is very small(such as 208 bytes).
I find that the H264 stream in the MP4 file is not stored in Annex-B format, so it can't find start code(0x000001), now my problem is how can I change the H264 stream to Annex-B format, and make it work?
I added start code at the beginning of every frame manually, but it still not work.
Anyone can give me any hint?Thanks very much.
Following is the codes similar with my:
// write the stream header, if any
av_write_header(pFormatCtxEnc);
.........
/**
* Init of Encoder and Decoder
*/
bool KeyFlag = false;
bool KeyFlagEx = false;
// Read frames and save frames to disk
int iPts = 1;
av_init_packet(&packet);
while(av_read_frame(pFormatCtxDec, &packet)>=0)
{
if (packet.flags == 1)
KeyFlag = true;
if (!KeyFlag)
continue;
if (m_bStop)
{
break;
}
// Is this a packet from the video stream?
if(packet.stream_index == videoStream)
{
currentframeNum ++;
if (progressCB != NULL && currentframeNum%20 == 0)
{
float fpercent = (float)currentframeNum/frameNum;
progressCB(fpercent,m_pUser);
}
if (currentframeNum >= beginFrame && currentframeNum <= endFrane)
{
if (packet.flags == 1)
KeyFlagEx = true;
if (!KeyFlagEx)
continue;
packet.dts = iPts ++;
av_write_frame(pFormatCtxEnc, &packet);
}
}
// Free the packet that was allocated by av_read_frame
}
// write the trailer, if any
av_write_trailer(pFormatCtxEnc);
/**
* Release of encoder and decoder
*/
return true;
You might try this: libavcodec/h264_mp4toannexb_bsf.c. It converts bitstream without start codes to bitstream with start codes.
Using your source file, does ffmpeg -i src.mp4 -vcodec copy -an dst.mp4 work? Does it work if you add -bsf h264_mp4toannexb? (all using the same version/build of ffmpeg as you are trying to use programmatically of course)

Resources