How to debug why Mac OS is not using Hardware H264 encoder - macos

I'm trying to encode some video only stream using H264, and I'm willing to use the hardware encoder in order to compare both quality and resource consumption between hardware and CPU encoding. The thing is that I'm not being able to force the OS to use the hardware encoder.
This is the code I'm using to create the VTCompressionSession:
var status: OSStatus
let encoderSpecifications: CFDictionary? = [
kVTVideoEncoderSpecification_EnableHardwareAcceleratedVideoEncoder as String: true,
kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder as String: true,
kVTVideoEncoderSpecification_EncoderID as String: "com.apple.videotoolbox.videoencoder.24rgb" // Tried without this paramenter so the system can decide what encoder ID should be using but doesn't work anyway.
]
let pixelBufferOptions: CFDictionary? = [
kCVPixelBufferWidthKey as String: Int(width),
kCVPixelBufferHeightKey as String: Int(height),
kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_24RGB) // Tried commenting this in case that there was a pixelformat constraint but didn't change anything
];
status = VTCompressionSessionCreate(kCFAllocatorDefault, width, height, CMVideoCodecType(kCMVideoCodecType_H264), encoderSpecifications, pixelBufferOptions, nil, { (outputCallbackRefCon: UnsafeMutablePointer<Void>, sourceFrameRefCon: UnsafeMutablePointer<Void>, status: OSStatus, infoFlags: VTEncodeInfoFlags, sampleBuffer: CMSampleBuffer?) -> Void in
...
}, unsafeBitCast(self, UnsafeMutablePointer<Void>.self), &compressionSession)
I opened the Console and this the only relevant message I'm getting when I try to create the session:
10/28/15 22:06:27.711 Dupla-Mac[87762]: <<<< VTVideoEncoderSelection >>>> VTSelectAndCreateVideoEncoderInstanceInternal: no video encoder found for 'avc1'
This is the status code I get when I use the EncoderID:
2015-10-28 22:17:13.480 Dupla-Mac[87895:5578917] Couldn't create compression session :( -12908
And this is the one I get when I don't use the EncoderID:
2015-10-28 22:18:16.695 Dupla-Mac[87996:5581914] Couldn't create compression session :( -12915
Both relate to the lack of availability of the resource, but couldn't find any difference. I've checked that the most known functionality that may use the hardware encoder are turned off, but I don't know how to check this for sure. AirPlay is off, QuickTime is off, there's not any app accessing the camera, and so.
TL;DR: is there any way to force or to know what's the strategy the OS is using to enable the Hardware Encoder, and eventually know why it is not available at any moment?
Thanks in advance!

I guess you've already resolved the problem but for others - the only HW-accelerated encoder available on macOS (10.8-10.12 for all macs 2012+) / iOS(8-10) is com.apple.videotoolbox.videoencoder.h264.gva and here is the full list: https://gist.github.com/vade/06ace2e33561a79cc240

get codec list
CFArrayRef encoder_list;
VTCopyVideoEncoderList(NULL, &encoder_list);
CFIndex size = CFArrayGetCount(encoder_list);
for(CFIndex i = 0; i < size; i++) {
CFDictionaryRef encoder_dict = CFArrayGetValueAtIndex(encoder_list, i);
CFStringRef type = CFDictionaryGetValue(encoder_dict, kVTVideoEncoderList_CodecType);
CFStringRef encoderID = CFDictionaryGetValue(encoder_dict, kVTVideoEncoderList_EncoderID);
CFStringRef codecName = CFDictionaryGetValue(encoder_dict, kVTVideoEncoderList_CodecName);
CFStringRef encoderName = CFDictionaryGetValue(encoder_dict, kVTVideoEncoderList_EncoderName);
CFStringRef display_name = CFDictionaryGetValue(encoder_dict, kVTVideoEncoderList_DisplayName);
NSLog(#" %# %# %# %# %#", type, encoderID, codecName, encoderName, display_name);
}

Related

vImage on the Mac: unable to convert from Core Video

Edit: Fixed. A working sample is at https://github.com/halmueller/vImage-mac-sample.
I'm trying to read the feed of a MacBook Pro's Facetime camera to process it with the vImage framework. I'm following the example in Apple's VideoCaptureSample, which is written for iOS.
I'm getting hung up on creating the vImageConverter, which creates an image buffer that vImage can use. My call to vImageConverter_CreateForCVToCGImageFormat() fails, with the console error "insufficient information in srcCVFormat to decode image. vImageCVImageFormatError = -21601".
The same call works on iOS. But the image formats are different on iOS and macOS. On iOS, the vImageConverter constructor is able to infer the format information, but on macOS, it can't.
Here's my setup code:
func displayEqualizedPixelBuffer(pixelBuffer: CVPixelBuffer) {
var error = kvImageNoError
if converter == nil {
let cvImageFormat = vImageCVImageFormat_CreateWithCVPixelBuffer(pixelBuffer).takeRetainedValue()
let deviceRGBSpace = CGColorSpaceCreateDeviceRGB()
let dcip3SolorSpace = CGColorSpace(name: CGColorSpace.dcip3)
vImageCVImageFormat_SetColorSpace(cvImageFormat,
deviceRGBSpace)
print(cvImageFormat)
if let unmanagedConverter = vImageConverter_CreateForCVToCGImageFormat(
cvImageFormat,
&cgImageFormat,
nil,
vImage_Flags(kvImagePrintDiagnosticsToConsole),
&error) {
guard error == kvImageNoError else {
return
}
converter = unmanagedConverter.takeRetainedValue()
} else {
return
}
}
When I run on iOS, I see in the console:
vImageCVFormatRef 0x101e12210:
type: '420f'
matrix:
0.29899999499321 0.58700001239777 0.11400000005960
-0.16873589158058 -0.33126410841942 0.50000000000000
0.50000000000000 -0.41868758201599 -0.08131241053343
chroma location: <RGB Base colorspace missing>
RGB base colorspace: =Bo
On macOS, though, the call to vImageConverter_CreateForCVToCGImageFormat returns nil, and I see:
vImageCVFormatRef 0x10133a270:
type: '2vuy'
matrix:
0.29899999499321 0.58700001239777 0.11400000005960
-0.16873589158058 -0.33126410841942 0.50000000000000
0.50000000000000 -0.41868758201599 -0.08131241053343
chroma location: <RGB Base colorspace missing>
RGB base colorspace: Рü
2018-03-13... kvImagePrintDiagnosticsToConsole: vImageConverter_CreateForCVToCGImageFormat error:
insufficient information in srcCVFormat to decode image. vImageCVImageFormatError = -21601
Note that the image type (4 letter code) is different, as is the RGB base colorspace. I've tried on the Mac using dcip3ColorSpace instead of deviceRGB, and the results are the same.
What am I missing to get this vImageConverter created?
The -21601 error code means that the source CV format is missing chroma siting information (see http://dougkerr.net/Pumpkin/articles/Subsampling.pdf for a nice background of chroma siting). You can fix this by explicitly setting it with vImageCVImageFormat_SetChromaSiting. So, immediately after setting the format's color space, and before creating the converter (i.e. where you have print(cvImageFormat)), add the following:
vImageCVImageFormat_SetChromaSiting(cvImageFormat,
kCVImageBufferChromaLocation_Center)
Cheers!
simon
So, while the answer about calling setting the chorma property on the vImage format does work, there is a better way to do that. Just set the property on the core video pixel buffer and then when you call vImageCVImageFormat_CreateWithCVPixelBuffer() it will just work, like so:
NSDictionary *pbAttachments = #{
(__bridge NSString*)kCVImageBufferChromaLocationTopFieldKey:(__bridge NSString*)kCVImageBufferChromaLocation_Center,
(__bridge NSString*)kCVImageBufferAlphaChannelIsOpaque: (id)kCFBooleanTrue,
};
CVBufferRef pixelBuffer = cvPixelBuffer;
CVBufferSetAttachments(pixelBuffer, (__bridge CFDictionaryRef)pbAttachments, kCVAttachmentMode_ShouldPropagate);
For extra points, you can also set the colorspace ref with the kCVImageBufferICCProfileKey and the CGColorSpaceCopyICCData() API.

Add a chapter track while creating a video with AVFoundation

I'm creating a video (QuickTime .mov format, H.264 encoded) from a bunch of still images, and I want to add a chapter track in the process. The video is being created fine, and I am not detecting any errors, but QuickTime Player does not show any chapters. I am aware of this question but it does not solve my problem.
The old QuickTime Player 7, unlike recent versions, can show information about the tracks of a movie. When I open a movie with working chapters (created using old QuickTime code), I see a video track and a text track, and the video track knows that the text track is providing chapters for the video. Whereas, if I examine a movie created by my new code, there is a metadata track along with the video track, but QuickTime does not know that the metadata track is supposed to be providing chapters. Things I've read have led me to believe that one is supposed to use metadata for chapters, but has anyone actually gotten that to work? Would a text track work?
Here's how I am creating the AVAssetWriterInput for the metadata.
// Make dummy AVMetadataItem to get its format
AVMutableMetadataItem* dummyMetaItem = [AVMutableMetadataItem metadataItem];
dummyMetaItem.identifier = AVMetadataIdentifierQuickTimeUserDataChapter;
dummyMetaItem.dataType = (NSString*) kCMMetadataBaseDataType_UTF8;
dummyMetaItem.value = #"foo";
AVTimedMetadataGroup* dummyGroup = [[[AVTimedMetadataGroup alloc]
initWithItems: #[dummyMetaItem]
timeRange: CMTimeRangeMake( kCMTimeZero, kCMTimeInvalid )] autorelease];
CMMetadataFormatDescriptionRef metaFmt = [dummyGroup copyFormatDescription];
// Make the input
AVAssetWriterInput* metaWriterInput = [AVAssetWriterInput
assetWriterInputWithMediaType: AVMediaTypeMetadata
outputSettings: nil
sourceFormatHint: metaFmt];
CFRelease( metaFmt );
// Associate metadata input with video input
[videoInput addTrackAssociationWithTrackOfInput: metaWriterInput
type: AVTrackAssociationTypeChapterList];
// Associate metadata input with AVAssetWriter
[writer addInput: metaWriterInput];
// Create a metadata adaptor
AVAssetWriterInputMetadataAdaptor* metaAdaptor = [AVAssetWriterInputMetadataAdaptor
assetWriterInputMetadataAdaptorWithAssetWriterInput: metaWriterInput];
P.S. I tried using a text track instead (an AVAssetWriterInput of type AVMediaTypeText) and QuickTime Player says the result is "not a movie". Not sure what I'm doing wrong.
I managed to use a text track to provide chapters. I spent an Apple developer tech support incident and was told that this is the right way to do it.
Setup:
I assume that the AVAssetWriter has been created, and an AVAssetWriterInput for the video track has been assigned to it.
The trickiest part here is creating the text format description. The docs say that CMTextFormatDescriptionCreateFromBigEndianTextDescriptionData takes as input a TextDescription structure, but neglects to say where that structure is defined. It is in Movies.h, which is in QuickTime.framework, which is no longer part of the Mac OS SDK. Thanks, Apple.
// Create AVAssetWriterInput
AVAssetWriterInput* textWriterInput = [AVAssetWriterInput
assetWriterInputWithMediaType: AVMediaTypeText
outputSettings: nil ];
textWriterInput.marksOutputTrackAsEnabled = NO;
// Connect input to writer
[writer addInput: textWriterInput];
// Mark the text track as providing chapter for the video
[videoWriterInput addTrackAssociationWithTrackOfInput: textWriterInput
type: AVTrackAssociationTypeChapterList];
// Create the text format description, which we will need
// when creating each sample.
CMFormatDescriptionRef textFmt = NULL;
TextDescription textDesc;
memset( &textDesc, 0, sizeof(textDesc) );
textDesc.descSize = OSSwapHostToBigInt32( sizeof(textDesc) );
textDesc.dataFormat = OSSwapHostToBigInt32( 'text' );
CMTextFormatDescriptionCreateFromBigEndianTextDescriptionData( NULL,
(const uint8_t*)&textDesc, sizeof(textDesc), NULL, kCMMediaType_Text,
&textFmt );
Writing a Sample:
CMSampleTimingInfo timing =
{
CMTimeMakeWithSeconds( endTime - startTime, timeScale ), // duration
CMTimeMakeWithSeconds( startTime, timeScale ),
kCMTimeInvalid
};
CMSampleBufferRef textSample = NULL;
CMPSampleBufferCreateWithText( NULL, (CFStringRef)theTitle, true, NULL, NULL,
textFmt, &timing, &textSample );
[textWriterInput appendSampleBuffer: textSample];
The function CMPSampleBufferCreateWithText is taken from the open source CoreMediaPlus.

Why does CMSampleBufferGetImageBuffer return NULL

I have built some code to process video files on OSX, frame by frame. The following is an extract from the code which builds OK, opens the file, locates the video track (only track) and starts reading CMSampleBuffers without problem. However each CMSampleBufferRef I obtain returns NULL when I try to extract the pixel buffer frame. There's no indication in iOS documentation as to why I could expect a NULL return value or how I could expect to fix the issue. It happens with all the videos on which I've tested it, regardless of capture source or CODEC.
Any help greatly appreciated.
NSString *assetInPath = #"/Users/Dave/Movies/movie.mp4";
NSURL *assetInUrl = [NSURL fileURLWithPath:assetInPath];
AVAsset *assetIn = [AVAsset assetWithURL:assetInUrl];
NSError *error;
AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:assetIn error:&error];
AVAssetTrack *track = [assetIn.tracks objectAtIndex:0];
AVAssetReaderOutput *assetReaderOutput = [[AVAssetReaderTrackOutput alloc]
initWithTrack:track
outputSettings:nil];
[assetReader addOutput:assetReaderOutput];
// Start reading
[assetReader startReading];
CMSampleBufferRef sampleBuffer;
do {
sampleBuffer = [assetReaderOutput copyNextSampleBuffer];
/**
** At this point, sampleBuffer is non-null, has all appropriate attributes to indicate that
** it's a video frame, 320x240 or whatever and looks perfectly fine. But the next
** line always returns NULL without logging any obvious error message
**/
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
if( pixelBuffer != NULL ) {
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
...
other processing removed here for clarity
}
} while( ... );
To be clear, I've stripped all error checking code but no problems were being indicated in that code. i.e. The AVAssetReader is reading, CMSampleBufferRef looks fine etc.
You haven't specified any outputSettings when creating your AVAssetReaderTrackOutput. I've run into your issue when specifying "nil" in order to receive the video track's original pixel format when calling copyNextSampleBuffer. In my app I wanted to ensure no conversion was happening when calling copyNextSampleBuffer for the sake of performance, if this isn't a big concern for you, specify a pixel format in the output settings.
The following are Apple's recommend pixel formats based on the hardware capabilities:
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
Because you haven't supplied any outputSettings you're forced to use the raw data contained within in the frame.
You have to get the block buffer from the sample buffer using CMSampleBufferGetDataBuffer(sampleBuffer), after you have that you need to get the actual location of the block buffer using
size_t blockBufferLength;
char *blockBufferPointer;
CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &blockBufferLength, &blockBufferPointer);
Look at *blockBufferPointer and decode the bytes using the frame header information for your required codec.
FWIW: Here is what official docs say for the return value of CMSampleBufferGetImageBuffer:
"Result is a CVImageBuffer of media data. The result will be NULL if the CMSampleBuffer does not contain a CVImageBuffer, or if the CMSampleBuffer contains a CMBlockBuffer, or if there is some other error."
Also note that the caller does not own the returned dataBuffer from CMSampleBufferGetImageBuffer, and must retain it explicitly if the caller needs to maintain a reference to it.
Hopefully this info helps.

How to set speaker configuration programatically using Core Audio API on Mac OS X?

I have a 7.1 channel audio output device and a custom kext to drive that. My custom application needs to send 7.1 rear channel audio data to the device but the device receives only 2 channel audio data. I checked "Configure Speaker" option in "Audio MIDI setup" application and it's set to stereo. When I set it to "7.1 Rear Surround" everything works fine. In my final product, I don't want the user to have to do all of this manually. So, the question is - Is there any Core Audio API or any other means of doing this programatically?
Ok, after playing around with some Core Audio APIs, finally I could get this done.
Get the AudioDeviceID:
AudioDeviceID audioDevice = getMyAwesomeDeviceID();
Create an AudioObjectPropertyAddress:
AudioObjectPropertyAddress propertyAddress;
propertyAddress.mSelector = kAudioDevicePropertyPreferredChannelLayout;
propertyAddress.mScope = kAudioDevicePropertyScopeOutput;
propertyAddress.mElement = kAudioObjectPropertyElementMaster;
Query if the audio object has this property:
AudioObjectHasProperty(audioDevice, &propertyAddress)
Get the data size of this property and create AudioChannelLayout:
UInt32 propSize(0);
AudioObjectGetPropertyDataSize(audioDevice, &propertyAddress, 0, NULL, &propSize);
AudioChannelLayout* layout = (AudioChannelLayout*)malloc(propSize);
Configure your AudioChannelLayout structure (eg: stereo layout):
AudioChannelLabel labels[2] = {kAudioChannelLabel_Right, kAudioChannelLabel_Left};
layout->mNumberChannelDescriptions = 2;
for (UInt32 i = 2; i < layout->mNumberChannelDescriptions; i++) {
layout->mChannelDescriptions[i].mChannelLabel = labels[i];
layout->mChannelDescriptions[i].mChannelFlags = kAudioChannelFlags_AllOff;
}
Set the AudioObject property data:
AudioObjectSetPropertyData(audioDevice, &propertyAddress, 0, NULL, propSize, layout);

Is the AudioFilePlayer audio unit sandbox compatible?

I've run into a problem using the AudioFilePlayer audio unit with app sandboxing enabled on OS X 10.8. I have an AUGraph with only two nodes, consisting of an AudioFilePlayer unit connected to a DefaultOutput unit. The goal (right now) is to simply play a single audio file. If sandboxing is not enabled, everything works fine. If I enable sandboxing, AUGraphOpen() returns error -3000 (invalidComponentID). If I remove the file player node from the AUGraph, the error goes away, which at least implies that the audio file player is causing the problem.
Here's the code I use to set the file player node up:
OSStatus AddFileToGraph(AUGraph graph, NSURL *fileURL, AudioFileInfo *outFileInfo, AUNode *outFilePlayerNode)
{
OSStatus error = noErr;
if ((error = AudioFileOpenURL((__bridge CFURLRef)fileURL, kAudioFileReadPermission, 0, &outFileInfo->inputFile))) {
NSLog(#"Could not open audio file at %# (%ld)", fileURL, (long)error);
return error;
}
// Get the audio data format from the file
UInt32 propSize = sizeof(outFileInfo->inputFormat);
if ((error = AudioFileGetProperty(outFileInfo->inputFile, kAudioFilePropertyDataFormat, &propSize, &outFileInfo->inputFormat))) {
NSLog(#"Couldn't get format of input file %#", fileURL);
return error;
}
// Add AUAudioFilePlayer node
AudioComponentDescription fileplayercd = {0};
fileplayercd.componentType = kAudioUnitType_Generator;
fileplayercd.componentSubType = kAudioUnitSubType_AudioFilePlayer;
fileplayercd.componentManufacturer = kAudioUnitManufacturer_Apple;
fileplayercd.componentFlags = kAudioComponentFlag_SandboxSafe;
if ((error = AUGraphAddNode(graph, &fileplayercd, outFilePlayerNode))) {
NSLog(#"AUAudioFilePlayer node not found (%ld)", (long)error);
return error;
}
return error;
}
Note that fileURL in the AudioFileOpenURL() call is a URL obtained from security scoped bookmark data, and is the URL to a file that has been dragged into the application by the user.
If I set the com.apple.security.temporary-exception.audio-unit-host sandboxing entitlement, when AUGraphOpen() is called, the user is prompted to lower security settings, and assuming they accept, playback again works fine (the sandbox is disabled).
So, this points to the AudioFilePlayer unit not being sandbox-safe/compatible. Is this true? It's difficult to believe that Apple wouldn't have fixed such an important part of the CoreAudio API to be sandbox compatible. Also note that I specify the kAudioComponentFlag_SandboxSafe flag in the description passed to AUGraphAddNode, and that call does not fail. Also, I can only find one reference to AudioFilePlayer not being sandbox-safe online, in the form of this post to the CoreAudio mailing list, and it didn't receive any replies. Perhaps I'm making some other subtle mistake that happens to cause a problem with sandboxing enabled, but not when it's off (I'm new to Core Audio)?

Resources