How to extract motion vectors from h264 without a full decode - ffmpeg

I'm want to extract the motion vector information from the encoded stream without performing a full decode of the frame. I'm using ffmpeg library and aware of extracting the information using the av_frame_get_side_data. Unfortunately, it does a full decode of the frame before providing the AV_FRAME_DATA_MOTION_VECTORS. My hunch is it should be possible to parse the h264 to extract out the motion vectors only, ignoring all the other data. Any direction/pointers will be very helpful!

Related

Decoding HECV with FFMpeg. Multiple slices I frame issue

I'm facing issue decoding camera with HEVC codec and RTSP transport (live555). I'm trying to decode frames with ffmpeg (avcodec_send_packet/avcodec_receive_frame) but it decodes only the first 1/3 of the picture and the others 2/3 stay green rectangle. Each frame is sent divided into 3 parts (slices) :
I-Frame is I-B-I
P-Frame is P-P-P
I suppose FFmpeg is able to deal with this because it agrees with HEVC specification.
Have I to "concatenate" 3 slices before send them to ffmpeg ?
Could you help me please?
I try to send all slices before receive them but this doesn't work.
FFmpeg's H264 decoder indeed needs full frames as input. You can't send it individual slices. You can concatenate them yourself, or use a bitstream filter / parser which will do it for you. In this case, manual concatenation will probably work fine.

How to determin the chroma subsampling of a particular MJPEG subtype in Media Foundation?

When reading available media types in Media Foundation is there any way of determining the chroma subsampling (i.e. 4:4:4 vs 4:2:0 etc) used for a particular MJPG format or do you have to extract that information yourself from the JPEG data once you start grabbing frames?
Short answer: you need to extract JPEG and then look into it.
Longer answer: Media Foundation does not deal with JPEGs exactly, and Motion JPEG as a sequence of JPEG images does not have to have such detail as chroma subsampling. It can even vary between frames and Motion JPEG in general is assumed to be good as long as individual frames are decodable.
It is possible that a demultiplexer reads subsampling information from track metadata or otherwise (including internally extracting that from first frame as you suggested) and then exposes it as a custom attribute on the MJPG video media type, but I doubt any of existing demultiplexers and especially stock ones is taking the trouble.

Extract motion vectors from x265 (HEVC) encoded video with ffmpeg/libavcodec?

I know that one can extract the motion vectors from an h264 encoded via by first setting the flag
av_dict_set(&opts, "flags2", "+export_mvs", 0);
then you can query the side-data for the motion vectors by doing this
sd = av_frame_get_side_data(frame, AV_FRAME_DATA_MOTION_VECTORS);
When I looked online to see if you can do something similar with HEVC encoded videos, I wasn't able to find any information. All I found was this by the definition of "AV_FRAME_DATA_MOTION_VECTORS"
Motion vectors exported by some codecs (on demand through the
export_mvs flag set in the libavcodec AVCodecContext flags2 option).
The data is the AVMotionVector struct defined in
libavutil/motion_vector.h.
but there was no information on exactly which codecs export this motion vector information. How would I go about finding this out?
If I'm not mistaken h264 is the only codec to print Motion Estimation Vectors.
I would suggest trying out the video filter mestimate.
Also, if you want to have a better ideia what's going on in ffmpeg, check the function ff_print_debug_info2 in libavcodec/mpegvideo.c

Decode/decompress H.264 back into raw/original file format, then encode into H.265

I have some files encoded using the H.264 codec.
There is a loss of quality when I convert them from H.264 to H.265.
I imagine I should convert them back to raw/original file format, then encode them into H.265.
Is it possible to decompress/decode H.264 into the original format (perhaps using FFMpeg)?
Is it the best way to convert from H.264 to H.265 without quality loss?
Thank you again for your help,
H.264 is lossy; the quality is lost at encoding time. There is no way to reconstruct the original from encoded form. In contrast, decoding is lossless - it produces exactly all of the information present in H.264 file, no more, no less. If your video editing software is not horrible, your H.264->H.265 conversion is the highest quality you can theoretically achieve given the compression settings you provide (without finding your original uncompressed file); there is no benefit in a separate decoding step, as that is what your software needs to do anyway.
Imagine a bad photocopy: there is no unphotocopier that can give you the original. That's what is happening with lossy compression.

Can the frame size be cropped during decoding using libavcodec?

I've followed Dranger's tutorial for displaying video using libav and FFMPEG. http://dranger.com/ffmpeg/
avcodec_decode_video2 seems to be the slowest part of the video decoding process. I will occasionally have two videos decoding simultaneously but only displaying half of each video side by side. In other words, half of each video will be off-screen. In order to speed up decoding, is there a way to only decode a portion of a frame?
No.
Codecs using interframe prediction need whole reference frames, so there's no way this could possibly work.

Resources