I have this following render callback for a remoteIO audio unit. Simply accessing the 0th element of the ioData parameter results in a crash. Very simply put, this works with no headphone jack connection but as soon as I plug a jack into my iphone6+, I get a bad access error when accessing the buffer.
If I plug it in while the app is running it crashes. If I plug it in first, and then build and run the app, it still crashes. I checked to see if inNumberFrames perhaps is changing based on a line out connection but it remains consistently at 512 frames.
OSStatus playbackCallback(void * inRefCon,
AudioUnitRenderActionFlags * ioActionFlags,
const AudioTimeStamp * inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData) {
float * output = (float*)ioData->mBuffers[0].mData;
output[0] = 1;
return noErr;
}
Apparently an AVAudioSession route change callback is called even if the headphones are plugged in before app launch. One of the things I tried was to delay remoteIO start until that point. However, the following code produces an error:
- (void)handleRouteChange:(NSNotification *)notification {
OSStatus err = AudioOutputUnitStart(remoteIOUnit);
}
The error:
Error: should alloc (-10849)
If ioData is nil in an Audio Unit callback, you need to have allocated your own AudioBufferList and buffer data memory (* mData), and use that memory instead. Some audio routes may have their own buffers, but this is not guaranteed. So your code needs to account for the case of a nil ioData.
Another possible cause of a nil ioData parameter is that your callback is an input callback, not an output callback, due to a bug in setting the callback for the correct RemoteIO bus.
The -10849 error might mean you are trying to start an audio unit that is already started and not stopped yet (it takes awhile after any audio stop command).
Related
AVFormatContext's interrupt_callback field is a
Custom interrupt callbacks for the I/O layer.
It's type is AVIOInterruptCB, and it explains in comment section:
Callback for checking whether to abort blocking functions.
AVERROR_EXIT is returned in this case by the interrupted function. During blocking operations, callback is called with opaque as parameter. If the callback returns 1, the blocking operation will be aborted.
No members can be added to this struct without a major bump, if new elements have been added after this struct in AVFormatContext or AVIOContext.
I have 2 questions:
what does the last section means? Especially "without a major bump"?
If I use this along with an RTSP source, when I close the input by avformat_close_input, the "TEARDOWN" message is being sent out, however it won't reach the RTSP server.
For 2: here is a quick pseudo-code for demo:
int pkts = 0;
bool early_exit = false;
int InterruptCallback(void* ctx) {
return early_exit ? 1 : 0;
}
void main() {
ctx = avformat_alloc_context
ctx->interrupt_callback.callback = InterruptCallback;
avformat_open_input
avformat_find_stream_info
pkts=0;
while(!early_exit) {
av_read_frame
if (pkts++ > 100) early_exit=true;
}
avformat_close_input
}
In case I don't use the interrupt callback at all, TEARDOWN is being sent out, and it also reaches the RTSP server so it can actually tear down the connection. Otherwise, it won't tear down it, and I have to wait until TCP socket times out.
What is the proper way of using this interrupt callback?
It means that they are not going to change anything for this structure (AVIOInterruptCB). However, if thats the case it would be in a major bump (major change from 4.4 eg to 5.0)
You need to pass a meaningful parameter to void* ctx. Anything that you like so you can check it within the static function. For example a bool that you will set as cancel so you will interrupt the av_read_frame (which will return an AVERROR_EXIT). Usually you pass a class of your decoder context or something similar which also holds all the info that you required to check whether to return 1 to interrupt or 0 to continue the requests properly. A real example would be that you open a wrong rtsp and then you want to open another one (the right one) so you need to cancel your previous requests.
I need to play raw PCM data (16 bit signed) using CoreAudio on OS X. I get it from network using UDP socket (on sender side data is captured from microphone).
The problem is that all I hear now is some short cracking noise and then only silence.
I'm trying to play data using AudioQueue. I setup it like this:
// Set up stream format fields
AudioStreamBasicDescription streamFormat;
streamFormat.mSampleRate = 44100;
streamFormat.mFormatID = kAudioFormatLinearPCM;
streamFormat.mFormatFlags = kLinearPCMFormatFlagIsBigEndian | kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;
streamFormat.mBitsPerChannel = 16;
streamFormat.mChannelsPerFrame = 1;
streamFormat.mBytesPerPacket = 2 * streamFormat.mChannelsPerFrame;
streamFormat.mBytesPerFrame = 2 * streamFormat.mChannelsPerFrame;
streamFormat.mFramesPerPacket = 1;
streamFormat.mReserved = 0;
OSStatus err = noErr;
// create the audio queue
err = AudioQueueNewOutput(&streamFormat, MyAudioQueueOutputCallback, myData, NULL, NULL, 0, &myData->audioQueue);
if (err)
{ PRINTERROR("AudioQueueNewOutput"); myData->failed = true; result = false;}
// allocate audio queue buffers
for (unsigned int i = 0; i < kNumAQBufs; ++i) {
err = AudioQueueAllocateBuffer(myData->audioQueue, kAQBufSize, &myData->audioQueueBuffer[i]);
if (err)
{ PRINTERROR("AudioQueueAllocateBuffer"); myData->failed = true; break; result = false;}
}
// listen for kAudioQueueProperty_IsRunning
err = AudioQueueAddPropertyListener(myData->audioQueue, kAudioQueueProperty_IsRunning, MyAudioQueueIsRunningCallback, myData);
if (err)
{ PRINTERROR("AudioQueueAddPropertyListener"); myData->failed = true; result = false;}
MyAudioQueueOutputCallback is:
void MyAudioQueueOutputCallback(void* inClientData,
AudioQueueRef inAQ,
AudioQueueBufferRef inBuffer)
{
// this is called by the audio queue when it has finished decoding our data.
// The buffer is now free to be reused.
MyData* myData = (MyData*)inClientData;
unsigned int bufIndex = MyFindQueueBuffer(myData, inBuffer);
// signal waiting thread that the buffer is free.
pthread_mutex_lock(&myData->mutex);
myData->inuse[bufIndex] = false;
pthread_cond_signal(&myData->cond);
pthread_mutex_unlock(&myData->mutex);
}
MyAudioQueueIsRunningCallback is:
void MyAudioQueueIsRunningCallback(void* inClientData,
AudioQueueRef inAQ,
AudioQueuePropertyID inID)
{
MyData* myData = (MyData*)inClientData;
UInt32 running;
UInt32 size;
OSStatus err = AudioQueueGetProperty(inAQ, kAudioQueueProperty_IsRunning, &running, &size);
if (err) { PRINTERROR("get kAudioQueueProperty_IsRunning"); return; }
if (!running) {
pthread_mutex_lock(&myData->mutex);
pthread_cond_signal(&myData->done);
pthread_mutex_unlock(&myData->mutex);
}
}
and MyData is:
struct MyData
{
AudioQueueRef audioQueue; // the audio queue
AudioQueueBufferRef audioQueueBuffer[kNumAQBufs]; // audio queue buffers
AudioStreamPacketDescription packetDescs[kAQMaxPacketDescs]; // packet descriptions for enqueuing audio
unsigned int fillBufferIndex; // the index of the audioQueueBuffer that is being filled
size_t bytesFilled; // how many bytes have been filled
size_t packetsFilled; // how many packets have been filled
bool inuse[kNumAQBufs]; // flags to indicate that a buffer is still in use
bool started; // flag to indicate that the queue has been started
bool failed; // flag to indicate an error occurred
bool finished; // flag to inidicate that termination is requested
pthread_mutex_t mutex; // a mutex to protect the inuse flags
pthread_mutex_t mutex2; // a mutex to protect the AudioQueue buffer
pthread_cond_t cond; // a condition varable for handling the inuse flags
pthread_cond_t done; // a condition varable for handling the inuse flags
};
I'm sorry if I posted too much code - hope it helps anyone to understand what exactly I do.
Mostly my code based on this code which is version of AudioFileStreamExample from Mac Developer Library adapted to work with CBR data.
Also I looked at this post and tried AudioStreamBasicDescription desribed there. And tried to change my flags to Little or Big Endian. It didn't work.
I looked at some another posts here and in the other resources while finding similar problem, I checked the order of my PCM data, for example. I just can't post more than two links.
Please anyone help me to understand what I'm doing wrong! Maybe I should abandon this way and use Audio Units right away? I'm just very newbie in CoreAudio and hoped that mid-level of CoreAudio will help me to solve this problem.
P.S. Sorry for my English, I tried as I can.
I hope you've solved this one on your own already, but for the benefit of other people who are having this problem, I'll post up an answer.
The problem is most likely because once an Audio Queue is started, time continues moving forward, even if you stop enqueueing buffers. But when you enqueue a buffer, it is enqueued with a timestamp that is right after the previously enqueued buffer. This means that if you don't stay ahead of the where the audio queue is playing, you will end up enqueuing buffers with a timestamp in the past, therefore the audio queue will go silent and the isRunning property will still be true.
To work around this, you have a couple of options. The simplest in theory would be to never fall behind on submitting buffers. But since you are using UDP, there is no guarantee that you will always have data to submit.
Another option is that you can keep track of what sample you should be playing and submit an empty buffer of silence whenever you need to have a gap. This option works good if your source data has timestamps that you can can use to calculate how much silence you need. But ideally, you wouldn't need to do this.
Instead you should be calculating the timestamp for the buffer using the system time. Instead of AudioQueueEnqueueBuffer, you'll need to use AudioQueueEnqueueBufferWithParameters instead. You just need to make sure the timestamp is ahead of where the queue is currently at. You'll also have to keep track what the system time was when you started the queue, so you can calculate the correct timestamp for each buffer you are submitting. If you have timestamp values on your source data, you should be able to use them to calculate the buffer timestamps as well.
I'm currently using a TPCircularBuffer to synchronize decoding audio data from an external library (libxmp: http://xmp.sourceforge.net/) and playing it back via the AudioUnits API on OS X.
Mach semaphores are used to signal when the buffer needs to be refilled.
However, there seems to be a "gap" in the audio (and the audio seems to play more slowly than usual) when the semaphore is triggered.
Is there any lower-latency method of synchronization that can be used in this case?
A proof-of-concept is here: https://gist.github.com/douglas-carmichael/cda1117e42e917397ed7
This is the structure I am passing down to the callback:
struct StreamData
{
TPCircularBuffer ringBuffer;
semaphore_t semaphore;
int fillThreshold;
};
Here is how I am creating the semaphore:
// Initialize our semaphore
mach_port_t self = mach_task_self();
kern_return_t ret = semaphore_create(self, &ourStream.semaphore, SYNC_POLICY_FIFO, 0);
if (ret != KERN_SUCCESS)
{
NSLog(#"Semaphore creation failed. Error <%d, %s>", ret, mach_error_string(ret));
return 0;
}
Here is the playback loop:
// Start our playback loop
struct xmp_frame_info ourFrameInfo;
int err = true;
while (xmp_play_frame(myContext) == 0)
{
xmp_get_frame_info(myContext, &ourFrameInfo);
if (ourFrameInfo.loop_count > 0)
break;
/* Are we getting a buffer overrun? */
if (err != false)
{
err = TPCircularBufferProduceBytes(&ourStream.ringBuffer, ourFrameInfo.buffer, ourFrameInfo.buffer_size);
}
semaphore_wait(ourStream.semaphore);
}
Here is the rendering callback:
static OSStatus renderModuleCallback(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inBufferFrames,
AudioBufferList *ioData)
{
struct StreamData *ourStream = inRefCon;
/* Initialize our variable for how much is available */
int bytesAvailable = 0;
/* Grab the data from the circular buffer into a temporary buffer */
SInt16 *ourBuffer = TPCircularBufferTail(&ourStream->ringBuffer, &bytesAvailable);
/* Do we have enough data? */
/* Note: fillThreshold is the maximum output buffer size for the device. */
if (bytesAvailable < ourStream->fillThreshold)
{
semaphore_signal(ourStream->semaphore);
}
/* memcpy() the data to the audio output */
memcpy(ioData->mBuffers[0].mData, ourBuffer, bytesAvailable);
/* Clear that section of the buffer */
TPCircularBufferConsume(inRefCon, bytesAvailable);
return noErr;
}
There appear to be Apple DTS recommendations not do anything that could lock or allocate memory inside a short real-time Audio Unit render callback, possibly even including semaphores and mach signaling.
Instead, an app can repeatedly poll the lock-free circular fifo in another thread (not the render callback thread). Given that both the sample rate and size of fifo are known, a polling rate clearly faster than the fifo empty-to-threshold rate should work, be fairly efficient, and require no locks. Latency can be controlled by varying both the threshold level and corresponding polling rate. An repeating NSTimer or CADisplayLink (or Open GL frame render) timer may be suitable for polling.
Note that the circular buffer needs to be larger than the fill threshold so the the fill routine has sufficient time to work asynchronous to the audio callback. And, of course, the worse case fill rate has to be faster than the best case fifo emptying rate.
I'm creating a Mac OS X CoreAudio command-line program with proprietary rendering of some alphanumeric terminal input into a live audio signal, by means of AudioUnits, trying to stay as simple as possible. All works fine up to matching output sample rate.
As a starting point I'm using the Chapter 07 tutorial code of Addisson Wesley's "Learning Core Audio", CH07_AUGraphSineWave.
I initialize the AudioComponent "by the book":
void CreateAndConnectOutputUnit (MyGenerator *generator)
{
AudioComponentDescription theoutput = {0};
theoutput.componentType = kAudioUnitType_Output;
theoutput.componentSubType = kAudioUnitSubType_DefaultOutput;
theoutput.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent comp = AudioComponentFindNext (NULL, &theoutput);
if (comp == NULL) {
printf ("can't get output unit");
exit (-1);
}
CheckError (AudioComponentInstanceNew(comp, &generator->outputUnit),
"Couldn't open component for outputUnit");
AURenderCallbackStruct input;
input.inputProc = MyRenderProc;
input.inputProcRefCon = generator;
CheckError(AudioUnitSetProperty(generator->outputUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
0,
&input,
sizeof(input)),
"AudioUnitSetProperty failed");
CheckError (AudioUnitInitialize(generator->outputUnit),
"Couldn't initialize output unit");
}
My main problem is in my not knowing how to retreive the output hardware sample rate for the rendering AURenderCallbackStruct
since it does play a vital part in the signal generating process. I can't afford having the sample rate hard-coded into the rendering callback, although knowing it's the easiest way, since rate mismatch causes the signal being played at wrong pitch.
Is there a way of getting the default output's sample rate on such a low-level API?
Is there a way of matching it somehow, without getting overly complicated?
Have I missed something?
Thanks in advance.
Regards,
Tom
When calling AudioUnitGetProperty, the 6th parameter must be a pointer to a variable that will get the size of the answer.
Float64 sampleRate;
UInt32 sampleRateSize;
CheckError(AudioUnitGetProperty(generator->outputUnit,
kAudioUnitProperty_SampleRate,
kAudioUnitScope_Input,
0,
&sampleRate,
&sampleRateSize),
"AudioUnitGetProperty failed");
However, as long as the sample rate has not been set, the function does not return a value (but there is also no error!)
You can however set the sample rate with for instance:
Float64 sampleRate = 48000;
CheckError(AudioUnitSetProperty(generator->outputUnit,
kAudioUnitProperty_SampleRate,
kAudioUnitScope_Input,
0,
&sampleRate,
sizeof(sampleRate)),
"AudioUnitGetProperty failed");
From now on you can also read the value with the Get-call.
This does not answer the question, what the default value is. As far as I know that is always 44100 Hz.
The sample-rate is a property of all AudioUnits - see kAudioUnitProperty_SampleRate (documentation here) - although ultimately it's the IO Unit (RemoteIO on iOS or HAL unit on MacOSX) that drives the sample-rate at the audio interface. This is not available in the call-back structure; you need to read this property with AudioUnitGetProperty() in your initialisation code.
In your case, the following would probably do it:
Float64 sampleRate;
CheckError(AudioUnitGetProperty(generator->outputUnit,
kAudioUnitProperty_SampleRate,
kAudioUnitScope_Input,
0,
&sampleRate,
sizeof(sampleRate)),
If you're targeting iOS, you also need to interact with the Audio Session.
I suppose I actually have two separate questions, but I think that they are related enough to include them both. The context is a Linux USB device driver (not userspace).
After transmitting a request URB, how do I receive the response once my complete callback is called?
How can I use interrupt URBs for single request/response pairs, and not as actual continuous interrupt polling (as they are intended)?
So for some background, I'm working on a driver for the Microchip MCP2210 a USB-to-SPI Protocol Converter with GPIO (USB 2.0, datasheet here). This device advertises as generic HID and exposes two interrupt endpoints (an in and an out) as well as it's control endpoint.
I am starting from a working, (but alpha-quality) demo driver written by somebody else and kindly shared with the community. However, this is a HID driver and the mechanism it uses to communicate with the device is very expensive! (sending a 64 byte message requires allocating a 6k HID report struct, and allocation is sometimes performed in the context of an interrupt, requiring GFP_ATOMIC!). We'll be accessing this from an embedded low-memory device.
I'm new to USB drivers and still pretty green with Linux device drivers in general. However, I'm trying to convert this to a plain-jane USB driver (not HID) so I can use the less expensive interrupt URBs for my communications. Here is my code for transmitting my request. For the sake of (attempted) brevity, I'm not including the definition of my structs, etc, but please let me know if you need more of my code. dev->cur_cmd is where I'm keeping the current command I'm processing.
/* use a local for brevity */
cmd = dev->cur_cmd;
if (cmd->state == MCP2210_CMD_STATE_NEW) {
usb_fill_int_urb(dev->int_out_urb,
dev->udev,
usb_sndintpipe(dev->udev, dev->int_out_ep->desc.bEndpointAddress),
&dev->out_buffer,
sizeof(dev->out_buffer), /* always 64 bytes */
cmd->type->complete,
cmd,
dev->int_out_ep->desc.bInterval);
ret = usb_submit_urb(dev->int_out_urb, GFP_KERNEL);
if (ret) {
/* snipped: handle error */
}
cmd->state = MCP2210_CMD_STATE_XMITED;
}
And here is my complete fn:
/* note that by "ctrl" I mean a control command, not the control endpoint */
static void ctrl_complete(struct urb *)
{
struct mcp2210_device *dev = urb->context;
struct mcp2210_command *cmd = dev->cur_cmd;
int ret;
if (unlikely(!cmd || !cmd->dev)) {
printk(KERN_ERR "mcp2210: ctrl_complete called w/o valid cmd "
"or dev\n");
return;
}
switch (cmd->state) {
/* Time to rx the response */
case MCP2210_CMD_STATE_XMITED:
/* FIXME: I think that I need to check the response URB's
* status to find out if it was even transmitted or not */
usb_fill_int_urb(dev->int_in_urb,
dev->udev,
usb_sndintpipe(dev->udev, dev->int_in_ep->desc
.bEndpointAddress),
&dev->in_buffer,
sizeof(dev->in_buffer),
cmd->type->complete,
dev,
dev->int_in_ep->desc.bInterval);
ret = usb_submit_urb(dev->int_in_urb, GFP_KERNEL);
if (ret) {
dev_err(&dev->udev->dev,
"while attempting to rx response, "
"usb_submit_urb returned %d\n", ret);
free_cur_cmd(dev);
return;
}
cmd->state = MCP2210_CMD_STATE_RXED;
return;
/* got response, now process it */
case MCP2210_CMD_STATE_RXED:
process_response(cmd);
default:
dev_err(&dev->udev->dev, "ctrl_complete called with unexpected state: %d", cmd->state);
free_cur_cmd(dev);
};
}
So am I at least close here? Secondly, both dev->int_out_ep->desc.bInterval and dev->int_in_ep->desc.bInterval are equal to 1, will this keep sending my request every 125 microseconds? And if so, how do I say "ok, ty, now stop this interrupt". The MCP2210 offers only one configuration, one interface and that has just the two interrupt endpoints. (I know everything has the control interface, not sure where that fits into the picture though.)
Rather than spam this question with the lsusb -v, I'm going to pastebin it.
Typically, request/response communication works as follows:
Submit the response URB;
submit the request URB;
in the request completion handler, if the request was not actually sent, cancel the response URB and abort;
in the response completion handler, handle the response data.
All that asynchronous completion handler stuff is a big hassle if you have a single URB that is completed almost immediately; therefore, there is the helper function usb_interrupt_msg() which works synchronously.
URBs to be used for polling must be resubmitted (typically from the completion handler).
If you do not resubmit the URB, no polling happens.