I`m trying to create "virtual microphone", that should work "in front" of default input device/microphone. So when user select "virtual microphone" as input (in Audacity, for example) and starts to record sound - Audacity will receive samples from virtual device driver, that was taken by driver from real/default microphone. So "virtual microphone" is a kind of proxy device for real (default/built-in/etc) one. This is needed for later processing of microphone input on-the-fly.
So far i created virtual HAL device (based on NullAudio.c driver example from Apple), i can generate procedural sound for Audacity, but i still can not figure out the way to read data from real microphone (using its deviceID) from inside driver.
Is it ok to use normal recording as in usual app (via AudioUnits/AURemoteIO/AUHAL), etc? Or something like IOServices should be used?
Documentation states that
An AudioServerPlugIn operates in a limited environment. First and
foremost, an AudioServerPlugIn may not make any calls to the client
HAL API in the CoreAudio.framework. This will result in undefined
(but generally bad) behavior.
but it is not clear what API is "client" API and what is not, in regard of reading microphone data.
What kind of API can/should be used from virtual device driver for accessing real microphone data in realtime?
Related
I'm working on a HAL virtual audio device.
I'm having problems getting the correct buffer size from the virtual audio device to my application...
How would I implement the properties kAudioDevicePropertyBufferFrameSize or kAudioDevicePropertyBufferFrameSizeRange to my virtual HAL device...
How would I do if I want to implement them to the apple nullaudio example found here: https://developer.apple.com/documentation/coreaudio/creating_an_audio_server_driver_plug-in
I tried to add them to my device the sam way as kAudioDevicePropertyNominalSampleRate is added to the nullAudio.c example. but with no success...
You have to set kAudioDevicePropertyBufferFrameSize in your client application (using AudioObjectSetPropertyData).
You can't control the kAudioDevicePropertyBufferFrameSize property from an AudioServerPlugin. It's only used by client processes to set the size of the IO buffers their IO procs receive.
When several clients use your device at the same time, CoreAudio lets them all use different IO buffer sizes (which might not be multiples/factors of each other), so your plug-in has to handle buffers of various sizes.
Source: https://lists.apple.com/archives/coreaudio-api/2013/Mar/msg00152.html
I'm not completely sure, but as far as I can tell, you can't control kAudioDevicePropertyBufferFrameSizeRange from an AudioServerPlugin either.
Recently, I wanted to get my hands dirty with Core Audio, so I started working on a simple desktop app that will apply effects (eg. echo) on the microphone data in real-time and then the processed data can be used on communication apps (eg. Skype, Zoom, etc).
To do that, I figured that I have to create a virtual microphone, to be able to send processed (with the applied effects) data over communication apps. For example, the user will need to select this new microphone (virtual) device as Input Device in a Zoom call so that the other users in the call can hear her with her voiced being processed.
My main concern is that I need to find a way to "route" the voice data captured from the physical microphone (eg. the built-in mic) to the virtual microphone. I've spent some time reading the book "Learning Core Audio" by Adamson and Avila, and in Chapter 8 the author explains how to write an app that a) uses an AUHAL in order to capture data from the system's default input device and b) then sends the data to the system's default output using an AUGraph. So, following this example, I figured that I also need to do create an app that captures the microphone data only when it's running.
So, what I've done so far:
I've created the virtual microphone, for which I followed the NullAudio driver example from Apple.
I've created the app that captures the microphone data.
For both of the above "modules" I'm certain that they work as expected independently, since I've tested them with various ways. The only missing piece now is how to "connect" the physical mic with the virtual mic. I need to connect the output of the physical microphone with the input of the virtual microphone.
So, my questions are:
Is this something trivial that can be achieved using the AUGraph approach, as described in the book? Should I just find the correct way to configure the graph in order to achieve this connection between the two devices?
The only related thread I found is this, where the author states that the routing is done by
sending this audio data to driver via socket connection So other apps that request audio from out virtual mic in fact get this audio from user-space application that listen for mic at the same time (so it should be active)
but I'm not quite sure how to even start implementing something like that.
The whole process I did for capturing data from the microphone seems quite long and I was thinking if there's a more optimal way to do this. The book seems to be from 2012 with some corrections done in 2014. Has Core Audio changed dramatically since then and this process can be achieved more easily with just a few lines of code?
I think you'll get more results by searching for the term "play through" instead of "routing".
The Adamson / Avila book has an ideal play through example that unfortunately for you only works for when both input and output are handled by the same device (e.g. the built in hardware on most mac laptops and iphone/ipad devices).
Note that there is another audio device concept called "playthru" (see kAudioDevicePropertyPlayThru and related properties) which seems to be a form of routing internal to a single device. I wish it were a property that let you set a forwarding device, but alas, no.
Some informal doco on this: https://lists.apple.com/archives/coreaudio-api/2005/Aug/msg00250.html
I've never tried it but you should be able to connect input to output on an AUGraph like this. AUGraph is however deprecated in favour of AVAudioEngine which last time I checked did not handle non default input/output devices well.
I instead manually copy buffers from the input device to the output device via a ring buffer (TPCircularBuffer works well). The devil is in the detail, and much of the work is deciding on what properties you want and their consequences. Some common and conflicting example properties:
minimal lag
minimal dropouts
no time distortion
In my case, if output is lagging too much behind input, I brutally dump everything bar 1 or 2 buffers. There is some dated Apple sample code called CAPlayThrough which elegantly speeds up the output stream. You should definitely check this out.
And if you find a simpler way, please tell me!
Update
I found a simpler way:
create an AVCaptureSession that captures from your mic
add an AVCaptureAudioPreviewOutput that references your virtual device
When routing from microphone to headphones, it sounded like it had a few hundred milliseconds' lag, but if AVCaptureAudioPreviewOutput and your virtual device handle timestamps properly, that lag may not matter.
My team has been struggling with a pretty strange issue while using the WinRT/C++ APIs for Windows to connect to both a MIDI port and receive BLE notifications through a proprietary service on the same device.
The WinRT/C++ library itself is really nice and provides easy and modern C++ interfaces to access the managed Windows runtime classes.
I've pushed a sample repo to Github where we've replicated the issue with a minimal example.
The repo's readme goes over the problem in detail, but I'll post the relevant bits here for completeness.
The sample program is performing roughly these steps:
Check for available MIDI devices using a DeviceWatcher.
Check for available Bluetooth LE devices using another instance of a DeviceWatcher.
Match discovered MIDI and BluetoothLE devices on their ContainerId property (see DeviceInfo for details). This is the method JUCE employs in the native WinRT code for their library, and works as expected.
Open the MIDI port and attach a handler to the MessageReceived event (see the code).
This causes the system to create a connection to the Bluetooth LE device. The program detects this state change, creates a BluetoothLEDevice, we perform GATT service discovery and attach a handler to the ValueChanged event for the characteristic we're interested in notifications from (see the code).
The program then counts how many MIDI messages are received on each port and how many BLE notifications are received from the corresponding device.
The behaviour we notice is that data from the most recently connected device streams just fine, while the throughput for the others is severly limited. We are at quite a standstill regarding this issue, and are not sure where the problem may lie.
We are at quite a standstill here. I'd be more willing to accept it if all the devices would exhibit this behaviour, but that's not the case. Is there any reason that creating both a MidiInPort and an BluetoothLEDevice from the same peripheral should cause this issue?
A BLE radio can only receive or send at any given time. And therefore only communicate with one device at any given time. It uses a scheduler to allocate radio time for every device when you have many devices. That way a second connection can "interrupt" a connection event from another device, decreasing the throughput for that device. See https://infocenter.nordicsemi.com/topic/sds_s132/SDS/s1xx/multilink_scheduling/central_connection_timing.html
From a general standpoint, I am trying to figure out how to access a platform device from userspace. To be more specific, I have a EMIF controller on and SoC of which I have added to my device tree and I believe it is correctly bound to a pre-written EMIF platform device driver. Now I am trying to figure out how I can access this EMIF device from a userspace application. I have come accross a couple different topics that seem to have some connection to this issue but I cannot quite find out how they relate.
1) As I read it seems like most I/O is done through the use of device nodes which are created by mknod(), do I need to create a device node in order to access this device?
2) I have read a couple threads that talk about writting a Kernel module (Character?, Block?) that can interface with both userspace and the platform device driver, and use it as an intermediary.
3) I have read about the possibility of using mmap() to map the memory of my platform device into my virtual memory space. Is this possible?
4) It seems that when the EMIF driver is instantiated, it calls the probe() fucntion. What functions would a userpace application call in the driver?
It's not completely clear what you're needing to do (and I should caveat that I have no experience with EMIF or with "platform devices" specifically), but here's some overview to help you get started:
Yes, the usual way of providing access to a device is via a device node. Usually this access is provided by a character device driver unless there's some more specific way of providing it. Most of the time if an application is talking "directly" to your driver, it's a character device. Most other types of devices are used in interfacing with other kernel subsystems: for example, a block device is typically used to provide access from a file system driver (say) to an underlying disk drive; a network driver provides access to the network from the in-kernel TCP/IP stack, etc.
There are several char device methods or entry points that can be supported by your driver, but the most common are "read" (i.e. if a user-space program opens your device and does a read(2) from it), "write" (analogous for write(2)) and "ioctl" (often used for configuration/administrative tasks that don't fall naturally into either a read or write). Note that mknod(2) only creates the user-space side of the device. There needs to be a corresponding device driver in the kernel (the "major device number" given in the mknod call links the user-space node with the driver).
For actually creating the device node in the file system, this can be automated (i.e. the node will automatically show up in /dev) if you call the right kernel functions while setting up your device. There's a special daemon that gets notifications from the kernel and responds by executing the mknod(2) system call.
A kernel module is merely a dynamically loadable way of creating a driver or other kernel extension. It can create a character, block or network device (et al.), but then so can a statically linked module. There are some differences in capability mostly because not all kernel functions you might want to use are "exported" to (i.e. visible to) dynamically loaded modules.
It's possible to support mapping of the device memory into user virtual memory space. This would be implemented by yet another driver entry point (mmap). See struct file_operations for all the entry points a char driver can support.
This is pretty much up to you: it depends on what the application needs to be able to do. There are many drivers in the kernel that provide no direct function to user-space, only to other kernel code. As to "probe", there are many probe functions defined in various interfaces. In most cases, these are called by the kernel (or perhaps by a 'higher level "class" driver') to allow the specific driver to discover, identify and "claim" individual devices. They (probe functions) don't usually have anything directly to do with providing access from user-space but I might well be missing something in a particular interface.
You need to create a device node in order to access the device.
The probe function is called when the driver finds a matching device.
For information on platform device API, the following articles could be useful.
The platform device API
Platform devices and device trees
I am new to Linux and I need to write a USB driver for a device with 2 interfaces. One interface is HID class (3/0/0) with one interrupt in endpoint and a report descriptor. The other interface is vendor defined with 3 bulk endpoints. In my usb_device_id table I have a USB_DEVICE entry with the VID and PID.
When I plug in the device, my xxx_probe function is called for the vendor defined interface but not for the HID interface. Instead, it appears that a built-in driver called 'generic-usb' is taking control of the HID interface.
Is there a way to ensure that my driver probe function is called first?
Why doesn't Linux make multiple passes looking for a more specific driver first (like Windows does)?
Alternatively, can the 'generic-usb' driver be used to receive data on the interrupt endpoint and to set reports and features on the control pipe?
It appears that libusb-1.0.8 allows an application to take control of interfaces on an attached device without the need of a custom driver. So far it appears to provide all the support that I need.