I've run into a problem using the AudioFilePlayer audio unit with app sandboxing enabled on OS X 10.8. I have an AUGraph with only two nodes, consisting of an AudioFilePlayer unit connected to a DefaultOutput unit. The goal (right now) is to simply play a single audio file. If sandboxing is not enabled, everything works fine. If I enable sandboxing, AUGraphOpen() returns error -3000 (invalidComponentID). If I remove the file player node from the AUGraph, the error goes away, which at least implies that the audio file player is causing the problem.
Here's the code I use to set the file player node up:
OSStatus AddFileToGraph(AUGraph graph, NSURL *fileURL, AudioFileInfo *outFileInfo, AUNode *outFilePlayerNode)
{
OSStatus error = noErr;
if ((error = AudioFileOpenURL((__bridge CFURLRef)fileURL, kAudioFileReadPermission, 0, &outFileInfo->inputFile))) {
NSLog(#"Could not open audio file at %# (%ld)", fileURL, (long)error);
return error;
}
// Get the audio data format from the file
UInt32 propSize = sizeof(outFileInfo->inputFormat);
if ((error = AudioFileGetProperty(outFileInfo->inputFile, kAudioFilePropertyDataFormat, &propSize, &outFileInfo->inputFormat))) {
NSLog(#"Couldn't get format of input file %#", fileURL);
return error;
}
// Add AUAudioFilePlayer node
AudioComponentDescription fileplayercd = {0};
fileplayercd.componentType = kAudioUnitType_Generator;
fileplayercd.componentSubType = kAudioUnitSubType_AudioFilePlayer;
fileplayercd.componentManufacturer = kAudioUnitManufacturer_Apple;
fileplayercd.componentFlags = kAudioComponentFlag_SandboxSafe;
if ((error = AUGraphAddNode(graph, &fileplayercd, outFilePlayerNode))) {
NSLog(#"AUAudioFilePlayer node not found (%ld)", (long)error);
return error;
}
return error;
}
Note that fileURL in the AudioFileOpenURL() call is a URL obtained from security scoped bookmark data, and is the URL to a file that has been dragged into the application by the user.
If I set the com.apple.security.temporary-exception.audio-unit-host sandboxing entitlement, when AUGraphOpen() is called, the user is prompted to lower security settings, and assuming they accept, playback again works fine (the sandbox is disabled).
So, this points to the AudioFilePlayer unit not being sandbox-safe/compatible. Is this true? It's difficult to believe that Apple wouldn't have fixed such an important part of the CoreAudio API to be sandbox compatible. Also note that I specify the kAudioComponentFlag_SandboxSafe flag in the description passed to AUGraphAddNode, and that call does not fail. Also, I can only find one reference to AudioFilePlayer not being sandbox-safe online, in the form of this post to the CoreAudio mailing list, and it didn't receive any replies. Perhaps I'm making some other subtle mistake that happens to cause a problem with sandboxing enabled, but not when it's off (I'm new to Core Audio)?
Related
I have a really basic little command line app that grabs the mouse coordinates the next time the mouse is clicked.
#import <Foundation/Foundation.h>
#import <AppKit/AppKit.h>
CGEventRef myCGEventCallback(CGEventTapProxy proxy, CGEventType type, CGEventRef event, void *refcon) {
CGFloat displayScale = 1.0f;
if ([[NSScreen mainScreen] respondsToSelector:#selector(backingScaleFactor)])
{
displayScale = [NSScreen mainScreen].backingScaleFactor;
}
CGPoint loc = CGEventGetLocation(event);
CFRelease(event);
printf("%dx%d\n", (int)roundf(loc.x * displayScale), (int)roundf(loc.y * displayScale) );
exit(0);
return event;
}
int main(int argc, const char * argv[]) {
#autoreleasepool {
CFMachPortRef eventTap;
CGEventMask eventMask;
CFRunLoopSourceRef runLoopSource;
eventMask = 1 << kCGEventLeftMouseDown;
eventTap = CGEventTapCreate(kCGSessionEventTap, kCGHeadInsertEventTap,
1, eventMask, myCGEventCallback, #"mydata");
runLoopSource = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, eventTap, 0);
CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSource,
kCFRunLoopCommonModes);
CGEventTapEnable(eventTap, true);
CFRunLoopRun();
}
return 0;
}
I'm building it with cmake with the following file:
cmake_minimum_required(VERSION 3.0.0)
project (location)
set(CMAKE_C_FLAGS "-arch x86_64 -mmacosx-version-min=10.12 -std=gnu11 -fobjc-arc -fmodules")
This all worked fine until the upgrade to Mojave.
A bit of poking around shows this is down to the latest set of security updates and some hints (except CGEventTapCreate() is not returning null) about settings some values in Info.plist to allow the app to use the accessibility API. But I'm struggling to work out where to put it as I just have a single .m file with the code.
Edit
This needs to run as a none root user (company policy)
if the only way to get it to ask for permission then it can be extended to be a "GUI" app with a minimal UI
This app is just to grab the upper left hand corner of a region of the screen to feed to a second app that streams that area of screen to a second device. The code for the streamer is common across Win/Linux/MacOS so trying to keep the screen coordinate collection totally separate
As you surmise, event taps won't work on Mojave without having accessibility access. From the documentation:
Event taps receive key up and key down events if one of the following
conditions is true: The current process is running as the root user.
Access for assistive devices is enabled. In OS X v10.4, you can enable
this feature using System Preferences, Universal Access panel,
Keyboard view.
A GUI app will prompt the user to enable accessibility the first time it's needed, but it looks like a CLI app doesn't do that (which makes sense).
There is no way to enable this programatically or through a script; the user must do it themselves.
Running your tool as root should work - can you enforce that?
Otherwise, you can direct the user to the correct place in System Preferences:
tell application "System Preferences"
reveal anchor "Privacy_Accessibility" of pane id "com.apple.preference.security"
activate
end tell
It may be possible using Carbon, if your app isn't sandboxed.
Finally, a quick test shows this is at least possible using IOHID. I shameless borrowed the KeyboardWatcher class from this answer. Then, modified the device type:
[self watchDevicesOfType:kHIDUsage_GD_Keyboard];
into:
[self watchDevicesOfType:kHIDUsage_GD_Mouse];
Finally, my callback looks like this:
static void Handle_DeviceEventCallback (void *inContext, IOReturn inResult, void *inSender, IOHIDValueRef value)
{
IOHIDElementRef element = IOHIDValueGetElement(value);
IOHIDElementType elemType = IOHIDElementGetType(element);
if (elemType == kIOHIDElementTypeInput_Button)
{
int elementValue = (int) IOHIDValueGetIntegerValue(value);
// 1 == down 0 == up
if (elementValue == 1)
{
CGEventRef ourEvent = CGEventCreate(NULL);
CGPoint point = CGEventGetLocation(ourEvent);
printf("Mouse Position: %.2f, y = %.2f \n", (float) point.x, (float) point.y);
}
}
}
That is really a quick hack job, but it demonstrates this is possible and hopefully you can refine it to your needs.
I've found the CGEventTap documentation is out of date beginning with Mojave. Running as root used to act as a bypass for certain entitlements, but in Mojave this was tightened down. One bizarre side effect, as you noticed, is that root can still acquire the mach port for the tap; its just that no events can be read from it. If you try your application without running as root you should get the expected popup asking for permission.
If you do not get the popup, or need to run as root for other purposes, you can manually add your application to the trusted TCC database via SystemPreferences -> Security & Privacy -> Privacy -> Accessibility
settings some values in Info.plist to allow the app to use the accessibility API
I believe you mean adding entitlements (which are also a plist). The entitlement that allows an application to use the Accessibility API is the com.apple.private.tcc.allow entitlement (with a value of kTCCServiceAccessibility). As you can probably guess from the name it is only allowed on Apple signed binaries.
You can add these entitlements to your own app if you disable System Integrity Protection (SIP) and boot the kernel with the option amfi_get_out_of_my_way=1, but I wouldn't recommend it (and certainly any customers of yours wouldn't want to). With just SIP disabled you could manually add an entry to the TCC database to grant privileges, but still wouldn't recommend it.
Possible Alternative
You can use an event monitor:
NSEventMask mask = (NSLeftMouseDownMask | NSRightMouseDownMask | NSOtherMouseDownMask);
mouseEventMonitor = [NSEvent addGlobalMonitorForEventsMatchingMask: mask
handler:^(NSEvent *event){
// get the current coordinates with this
NSPoint coords = [NSEvent mouseLocation];
// event cooordinates would be event.absoluteX and event.absoluteY
... do stuff
}];
The documentation does mention:
Key-related events may only be monitored if accessibility is enabled or if your application is trusted for accessibility access (see AXIsProcessTrusted).
But I don't think that applies to mouse events.
I am trying to make a custom media sink for video playback in an OpenGL application (without the various WGL_NV_DX_INTEROP, as I am not sure if all my target devices support this).
What I have done so far is to write a custom stream sink that accepts RGB32 samples and set up playback with a media session, however i encountered a problem with initial testing of playing an mp4 file:
one (or more) of the MFTs in the generated topology keep failing with an error code MF_E_TRANSFORM_NEED_MORE_INPUT, therefore my stream sink never receives samples
After a few samples have been requested, the media session receives the event MF_E_ATTRIBUTENOTFOUND, but I still don't know where it is coming from
If, however, I configure the stream sink to receive NV12 samples, everything seems to work fine.
My best guess is the color converter MFT generated by the TopologyLoader needs some more configuration, but I don't know how to do that, considering that I need to keep this entire process indipendent from the original file types.
I've made a minimal test case, that demonstrate the use of a custom video renderer with a classical Media Session.
I use big_buck_bunny_720p_50mb.mp4, and i don't see any problems using RGB32 format.
Sample code here : https://github.com/mofo7777/Stackoverflow under MinimalSinkRenderer.
EDIT
Your program works well with big_buck_bunny_720p_50mb.mp4. I think that your mp4 file is the problem. Share it, if you can.
I just made a few changes :
You Stop on MESessionEnded, and you Close on MESessionStopped.
case MediaEventType.MESessionEnded:
Debug.WriteLine("MediaSession:SesssionEndedEvent");
hr = mediaSession.Stop();
break;
case MediaEventType.MESessionClosed:
Debug.WriteLine("MediaSession:SessionClosedEvent");
receiveSessionEvent = false;
break;
case MediaEventType.MESessionStopped:
Debug.WriteLine("MediaSession:SesssionStoppedEvent");
hr = mediaSession.Close();
break;
default:
Debug.WriteLine("MediaSession:Event: " + eventType);
break;
Adding this to wait for the sound, and to check sample is ok :
internal HResult ProcessSample(IMFSample s)
{
//Debug.WriteLine("Received sample!");
CurrentFrame++;
if (s != null)
{
long llSampleTime = 0;
HResult hr = s.GetSampleTime(out llSampleTime);
if (hr == HResult.S_OK && ((CurrentFrame % 50) == 0))
{
TimeSpan ts = TimeSpan.FromMilliseconds(llSampleTime / (10000000 / 1000));
Debug.WriteLine("Frame {0} : {1}", CurrentFrame.ToString(), ts.ToString());
}
// Do not call SafeRelease here, it is done by the caller, it is a parameter
//SafeRelease(s);
}
System.Threading.Thread.Sleep(26);
return HResult.S_OK;
}
In
public HResult SetPresentationClock(IMFPresentationClock pPresentationClock)
adding
SafeRelease(PresentationClock);
before
if (pPresentationClock != null)
PresentationClock = pPresentationClock;
We've been working with AudioUnits in Core Audio. It is simultaniously a very powerful audio framework, and one of the worst documented which makes it both a joy and a frustration to work with.
We want to accomplish something we know iPads had been able to do since iOS 6.0 - Multiple audio inputs.
So far - from the 2012 Developer Talk - It appears you have to set the audio session to MultiRoute. We've done this. If I plug in an a soundcard from a keyboard. I can see that there are two inputs. Great. We're then told that we need to set a ChannelMap on a Remote I/O unit.
To what? Well... here's where it gets vague. We need to set all the channels we don't want to -1 and the channels we want to 0 and 1 (for stereo input or for mono?).
We attempt this and... nothing. Sound still plays through on the 'last in wins' principle. Microphone if everything plugged out, soundcard if that's the one plugged in. But we can't switch between them.
This setup code is always run before the other function listed
func setupAudioSession() {
self.audioSession = AVAudioSession.sharedInstance()
do {
try audioSession.setCategory(AVAudioSessionCategoryMultiRoute, with: [.mixWithOthers])
try audioSession.setActive(true)
audioSessionWasSetup = true
} catch let error {
//TODO: Implement something here
print(error)
audioSessionWasSetup = false
}
}
We then have a remote I/O with an associated audiograph set up. This has been tested and works beautifully. But we need to be able to set where it's pulling sound from.
I've attempted to do it with this, but not only doesn't it have any effect... nothing happens.
Am I missing something?
private func setChannelMap(onAudioUnit audioUnit: AudioUnit?, toChannel channelIndex: Int = 0) {
var channelMap: [Int32] = []
if audioUnit == nil {
return
}
var numberOfInputChannels: UInt32 = 4 // Two stereo inputs? - I'm just guessing here
let mapSize: UInt32 = numberOfInputChannels * UInt32(MemoryLayout<Int32>.size);
for _ in 0...(numberOfInputChannels) {
channelMap.append(-1)
}
channelMap[2 * channelIndex] = 0;
channelMap[2 * channelIndex + 1] = 1;
let status = AudioUnitSetProperty(audioUnit!,
kAudioOutputUnitProperty_ChannelMap,
kAudioUnitScope_Input,
0,
&channelMap,
mapSize);
self.checkError(status, "Failed to set Channel Map on input unit")
}
There isn't any documentation on this at all as far as I've been able to find. Nor any code examples.
I hope you can help us.
I wrote a command line c tool generating an sine wave and playing it using CoreAudio on the default audio output. I am initializing a
AURenderCallbackStruct and initialize an AudioUnit using AudioUnitInitialize (as already discussed in this forum). All this is working as intended, but when it comes to closing the program I am not able to close the AudioUnit, neither with using AudioOutputUnitStop(player.outputUnit); nor AudioOutputUnitStop(player.outputUnit); nor
AudioComponentInstanceDispose(player.outputUnit);
The order of appearance of these calls in the code does not change the behavior.
The program is compiled without error messages, but the sine is still audible as long as the rest of the program is running.
Here is the code I'm using for initializing the AudioUnit:
void CreateAndConnectOutputUnit (ToneGenerator *player) {
AudioComponentDescription outputcd = {0};
outputcd.componentType = kAudioUnitType_Output;
outputcd.componentSubType = kAudioUnitSubType_DefaultOutput;
outputcd.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent comp = AudioComponentFindNext (NULL, &outputcd);
if (comp == NULL) {
printf ("can't get output unit");
exit (-1);
}
AudioComponentInstanceNew(comp, &player->outputUnit);
// register render callback
AURenderCallbackStruct input;
input.inputProc = SineWaveRenderCallback;
input.inputProcRefCon = player;
AudioUnitSetProperty(player->outputUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Output,
0,
&input,
sizeof(input);
// initialize unit
AudioUnitInitialize(player->outputUnit);
}
In my main program I'm starting the AudioUnit and the sine wave.
void main {
// code for doing various things
ToneGenerator player = {0}; // create a sound object
CreateAndConnectOutputUnit (&player);
AudioOutputUnitStart(player.outputUnit);
// waiting to listen to the sine wave
sleep(3);
// attempt to stop the sound output
AudioComponentInstanceDispose(player.outputUnit);
AudioUnitUninitialize(player.outputUnit);
AudioOutputUnitStop(player.outputUnit);
//additional code that should be executed without sine wave being audible
}
As I'm new to both, this forum as well as programming in Xcode I hope that I could explain this issue in a way that you can help me out and I hope that I didn't miss the answer somewhere in the forum while searching for a solution.
Thank you in advance for your time and input,
Stefan
You should manage and unmanage your audio unit in a logical order. It doesn't make sense to stop playback on an already uninitialized audio unit, which had in fact previously been disposed of in the middle of the playback. Rather than that, try the following order:
AudioOutputUnitStop(player.outputUnit); //first stops playback
AudioUnitUninitialize(player.outputUnit); //then deallocates unit's resources
AudioComponentInstanceDispose(player.outputUnit); //finally disposes of the AU itself
The sine wave command line app you're after is a well elaborated lesson in this textbook. Please read it step by step.
Last, but not least, your question has nothing to do with C++, CoreAudio is a plain-C API, so C++ in both your title and tag are wrong and misleading.
An Audio Unit runs in an asynchronous thread that may not actually stop immediately when you call AudioOutputUnitStop. Thus, it may work better to wait a fraction of a second (at least a couple audio callback buffer durations in time) before calling AudioUnitUninitialize and AudioComponentInstanceDispose on a potentially still running audio unit.
Also, check to make sure your player.outputUnit value is a valid unit (and not an uninitialized or trashed variable) at the time you stop the unit.
I have built some code to process video files on OSX, frame by frame. The following is an extract from the code which builds OK, opens the file, locates the video track (only track) and starts reading CMSampleBuffers without problem. However each CMSampleBufferRef I obtain returns NULL when I try to extract the pixel buffer frame. There's no indication in iOS documentation as to why I could expect a NULL return value or how I could expect to fix the issue. It happens with all the videos on which I've tested it, regardless of capture source or CODEC.
Any help greatly appreciated.
NSString *assetInPath = #"/Users/Dave/Movies/movie.mp4";
NSURL *assetInUrl = [NSURL fileURLWithPath:assetInPath];
AVAsset *assetIn = [AVAsset assetWithURL:assetInUrl];
NSError *error;
AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:assetIn error:&error];
AVAssetTrack *track = [assetIn.tracks objectAtIndex:0];
AVAssetReaderOutput *assetReaderOutput = [[AVAssetReaderTrackOutput alloc]
initWithTrack:track
outputSettings:nil];
[assetReader addOutput:assetReaderOutput];
// Start reading
[assetReader startReading];
CMSampleBufferRef sampleBuffer;
do {
sampleBuffer = [assetReaderOutput copyNextSampleBuffer];
/**
** At this point, sampleBuffer is non-null, has all appropriate attributes to indicate that
** it's a video frame, 320x240 or whatever and looks perfectly fine. But the next
** line always returns NULL without logging any obvious error message
**/
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
if( pixelBuffer != NULL ) {
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
...
other processing removed here for clarity
}
} while( ... );
To be clear, I've stripped all error checking code but no problems were being indicated in that code. i.e. The AVAssetReader is reading, CMSampleBufferRef looks fine etc.
You haven't specified any outputSettings when creating your AVAssetReaderTrackOutput. I've run into your issue when specifying "nil" in order to receive the video track's original pixel format when calling copyNextSampleBuffer. In my app I wanted to ensure no conversion was happening when calling copyNextSampleBuffer for the sake of performance, if this isn't a big concern for you, specify a pixel format in the output settings.
The following are Apple's recommend pixel formats based on the hardware capabilities:
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
Because you haven't supplied any outputSettings you're forced to use the raw data contained within in the frame.
You have to get the block buffer from the sample buffer using CMSampleBufferGetDataBuffer(sampleBuffer), after you have that you need to get the actual location of the block buffer using
size_t blockBufferLength;
char *blockBufferPointer;
CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &blockBufferLength, &blockBufferPointer);
Look at *blockBufferPointer and decode the bytes using the frame header information for your required codec.
FWIW: Here is what official docs say for the return value of CMSampleBufferGetImageBuffer:
"Result is a CVImageBuffer of media data. The result will be NULL if the CMSampleBuffer does not contain a CVImageBuffer, or if the CMSampleBuffer contains a CMBlockBuffer, or if there is some other error."
Also note that the caller does not own the returned dataBuffer from CMSampleBufferGetImageBuffer, and must retain it explicitly if the caller needs to maintain a reference to it.
Hopefully this info helps.