This is using a Phantom 4 Pro.
We have implemented a VideoFrameProcessor in our Swift app and register it with the DJIVideoPreviewer. This all seems to work great when we are in DJICameraPhotoAspectRatio.ratio16_9. The image shows up in the frame processor as 1280x720, and we can decode the data just fine into a UIImage.
However, if we switch to DJICameraPhotoAspectRatio.ratio4_3, then we get an image reported to be of size 960x720 and when we decode the data into an UIImage it has artifacts and green barring.
Is there a way to get proper images from the frame processor in 4:3 mode?
Here is 16:9, looks perfect.
Here is 4:3, has artifacts and green barring.
I finally was able to get hardware encoding working. That solved the issue. At some point in your code base you need to set a global function so DJI can load prebuilt I-frames for the h264 stream. It is not documented anywhere. Hope this helps someone else.
g_loadPrebuildIframeOverrideFunc = loadPrebuildIframePrivate
You can view the list of supported I frames here
You can see my GitHub comment here
Related
The mvhd atom or box of the original Quicktime MOV format supports a poster time variable for a timecode to use as a poster frame that can be used in preview scenarios as a thumbnail image or cover picture. As far as I can tell, the ISOBMFF-based MP4 format (.m4v) has inherited this feature, but I cannot find a way to set it using FFmpeg or MP4box or similar cross-platform CLI software. Edit: Actually, neither ISOBMFF nor MP4 imports this feature from MOV. Is there any other way to achieve this, e.g. using something like HEIFʼs derived images with a thmb (see Amendment 2) role?
The original Apple Quicktime (Pro) editor did have a menu option for doing just that. (Apple Compressor and Photos could do it, too).
To be clear, I do not want to attach a separate image file, which could possibly be a screenshot grabbed from a movie still, as a separate track to the multimedia container. I know how to do that:
Stackoverflow #54717175
Superuser #597945
I also know that some people used to copy the designated poster frame from its original position to the very first frame, but many automatically generated previews use a later time index, e.g. from 10 seconds, 30 seconds, 10% or 50% into the video stream.
I've written a screen-recording app that writes out H.264 movie files using VideoToolbox and AVWriter. The colors in the recorded files are a bit dull compared to the original screen. I know that this if because the colorProfile is not stored in the video-file.
This is closely related to How to color manage AVAssetWriter output
I have created a testbed to show this on github ScreenRecordTest
If you run this app you can start recording using CMD-R and stop it using the same CMD-R (You have to start and stop recording once to get a fully written movie-file). You will find the recording in your /tmp/ folder under a name similar like this: "/tmp/grab-2018-10-25 09:23:32 +0000.mov"
When recording the app shows two live images: a) the frame gotten from CGDisplayStream -and- b) the cmSampleBuffer that came out of the compressor.
What I found out is that the IOSurface that is returned from CGDisplayStream is not color-managed, so you'll notice the "dull" colors already before compression. If you un-comment line 89 in AppDelegate.swift
// cgImage = cgImage.copy(colorSpace: screenColorSpace)!
this live-preview will have the correct colors. Now this is only for displaying the IOSurface before compression. I have no idea how to make the other live-preview (after compression) (line 69 in AppDelegate) show correct colors (say: how to apply a colorProfile to a CMSampleBuffer) or most important how to tag the written video-file with the right profile so that when opening the .mov file I get the correct colors on playback.
I have ported Cobalt8 code onto our embedded system, and it is now able to decode VP9 up to 4k quality.
However, I ran into an issue with fast forward and rewind. Specially, when I fast forward a few times
and then followed by a rewind operation, there is a chance that audio or video streaming data will stop coming into
the range buffer. I am not familiar with the streaming mechanism. It would be great if someone can shed some light on
where or what I can look to debug this issue.
PS: I have drawn a quick picture to show the problem.
thanks.
Is it possible for you to tell me the values of the following settings in your configuration_public.h:
SB_MEDIA_SOURCE_BUFFER_STREAM_AUDIO_MEMORY_LIMIT
SB_MEDIA_SOURCE_BUFFER_STREAM_VIDEO_MEMORY_LIMIT
SB_MEDIA_MAIN_BUFFER_BUDGET
I am rendering a video file from input pictures that come from a 3D engine at runtime (I don't pass an actual picture file, just RGB memory).
This works perfectly when outputting MP4 using CODEC_ID_H264 as video codec.
But when I want to create an MPG file using CODEC_ID_MPEG2VIDEO, the resulting file is simply broken. No player can play the video correctly and when I then want to concatenate that MPG with another MPG file, and transform the result MP4 in another step, the resulting .mp4 file has both videos, but many frames from the original MPG video (and only video! Sound works fine) are simply skipped.
At first I thought the MPG -> MP4 conversion was the problem, but then I noticed that the initial MPG, which comes from the video render engine, is already broken, which would speak for broken headers. Not sure if it is the system or sequence headers that are broken, though.
Or if it could be something totally different.
If you want to have a look, here is the file:
http://www.file-upload.net/download-7093306/broken.mpg.html
Again, the exact same muxing code works perfectly fine when directly creating an MP4 from the video render engine, so I'm pretty sure the input data, swscale(), etc. is correct. The only difference is that CODEC_ID_H264 is used and some additional variables (like qmin, qmax, etc.) are set, which are all specific to H264 so should not have an impact.
Also, neither avformat_write_header nor av_write_trailer report an error.
As an additional info, when viewing the codec data of the MPG in VLC player, it is not able to show the FPS, resolution and format (should show 640x360, 30 fps and 4:2:0 YUV).
I am using a rather new (2-3 months old, maybe) FFmpeg version, which I compiled from sources with MinGW.
Any ideas on how to resolve this would be welcome. Currently, I am out of those :)
Alright, the problem was not the avformat_write_header, but that I did not set the PTS value of each written video packet to AV_NOPTS_VALUE.
Once I do set it for each video packet, everything works fine.
I assumed that AV_NOPTS_VALUE was the default, as I never needed to set any special PTS value.
Given a sample buffer of H.264, is there a way to extract the frame it represents as an image?
I'm using QTKit to capture video from a camera and using a QTCaptureMovieFileOutput as the output object.
I want something similar to the CVImageBufferRef that is passed as a parameter to the QTCaptureVideoPreviewOutput delegate method. For some reason, the file output doesn't contain the CVImageBufferRef.
What I do get is a QTSampleBuffer which, since I've set it in the compression options, contains an H.264 sample.
I have seen that on the iPhone, CoreMedia and AVFoundation can be used to create a CVImageBufferRef from the given CMSampleBufferRef (Which, I imagine is as close to the QTSampleBuffer as I'll be able to get) - but this is the Mac, not the iPhone.
Neither CoreMedia or AVFoundation are on the Mac, and I can't see any way to accomplish the same task.
What I need is an image (whether it be a CVImageBufferRef, CIImage or NSImage doesn't matter) from the current frame of the H.264 sample that is given to me by the Output object's call back.
Extended info (from the comments below)
I have posted a related question that focusses on the original issue - attempting to simply play a stream of video samples using QTKit: Playing a stream of video data using QTKit on Mac OS X
It appears not to be possible which is why I've moved onto trying to obtain frames as images and creating an appearance of video, by scaling, compressing and converting the image data from CVImageBufferRef to NSImage and sending it to a peer over the network.
I can use the QTCapturePreviewVideoOutput (or decompressed) to get uncompressed frame images in the form of CVImageBufferRef.
However, these images references need compressing, scaling and converting into NSImages before they're any use to me, hence the attempt to get an already scaled and compressed frame from the framework using the QTCaptureMovieFileOutput (which allows a compression and image size to be set before starting the capture), saving me from having to do the expensive compression, scale and conversion operations, which kill CPU.
Does the Creating a Single-Frame Grabbing Application section of the QTKit Application Programming Guide not work for you in this instance?