NSSound-like framework that works, but doesn't require dealing with a steep learning curve - cocoa

I've pretty much finished work on a white noise feature for one of my applications using NSSound to play a loop of 10 second AAC-encoded pre-recorded white noise.
[sound setLoops: YES]
should be all that's required, right?
It works like a charm but I've noticed that there is an audible pause between the sound file finishing and restarting.. a sort of "plop" sound. This isn't present when looping the original sound files and after an hour or so of trying to figure this out, I've come to the conclusion that NSSound sucks and that the audible pause is an artefact of the synchronisation of the private background thread playing the sound. It seems to be dependent on the main run loop somehow and this causes the audible gap between the end and restarting of the sound.
I know very little about sound stuff and this is a very minor feature, so I don't want to get into the depths of CoreAudio just to play a looping 10s sound fragment.. so I went chasing after a nice alternative, but nothing seems to quite fit:
Core Audio: total overkill, but at least a standard framework
AudioQueue: complicated, with C++ sample code!?
MusicKit/ SndKit: also huge learning curve, based on lots of open source stuff, etc.
I saw that AVFoundation on iOS 4 would be a nice way to play sounds, but that's only scheduled for Mac OS X 10.7..
Is there any easy-to-use way of reliably looping sound on Mac OS X 10.5+?
Is there any sample code for AudioQueue or Core Audio that takes the pain out of using them from an Objective-C application?
Any help would be very much appreciated..
Best regards,
Frank

Use QTKit. Create a QTMovie for the sound, set it to loop, and leave it playing.

Just for the sake of the archives.
QTKit also suffers from a gap between the end of one play through and start of the next one. It seems to be linked with re-initializing the data (perhaps re-reading it from disk?) in some way. It's a lot more noticeable when using the much smaller but highly compressed m4a format than when playing uncompressed aiff files but it's still there even so.
The solution I've found is to use Audio Queue Services:
http://developer.apple.com/mac/library/documentation/MusicAudio/Conceptual/AudioQueueProgrammingGuide/AQPlayback/PlayingAudio.html#//apple_ref/doc/uid/TP40005343-CH3-SW1
and
http://developer.apple.com/mac/library/samplecode/AudioQueueTools/Listings/aqplay_cpp.html#//apple_ref/doc/uid/DTS10004380-aqplay_cpp-DontLinkElementID_4
The Audio Queue calls a callback function which prepares and enqueues the next buffer, so when you reach the end of the current file you need to start again from the beginning. This gives completely gapless playback.
There's two gotchas in the sample code in the documentation.
The first is an actual bug (I'll contact DTS about this so they can correct it). Before allocating and priming the audio buffers, the custom structure must switch on playback otherwise the audio buffer never get primed and nothing is played:
aqData.mIsRunning = 1;
The second gotcha is that the code doesn't run in Cocoa but as a standalone tool, so the code connects the audio queue to a new run loop and actually implements the run loop itself as the last step of the program.
Instead of passing CFRunLoopGetCurrent(), just pass NULL which causes the AudioQueue to run in its own run loop.
result = AudioQueueNewOutput ( // 1
&aqData.mDataFormat, // 2
HandleOutputBuffer, // 3
&aqData, // 4
NULL, //CFRunLoopGetCurrent (), // 5
kCFRunLoopCommonModes, // 6
0, // 7
&aqData.mQueue // 8
);
I hope this can save the poor wretches trying to do this same thing in the future a bit of time :-)

Sadly, there is a lot of pain when developing audio applications on OS X. The learning curve is very steep because the documentation is fairly sparse.
If you don't mind Objective-C++ I've written a framework for this kind of thing: SFBAudioEngine. If you wanted to play a sound with my code here is how you could do it:
DSPAudioPlayer *player = new DSPAudioPlayer();
player->Enqueue((CFURLRef)audioURL);
player->Play();
Looping is also possible.

Related

Audio Unit (AUv3) in macOS only works in selected track in DAW

When I build an Audio Unit extension (instrument/aumu) using the template in XCode (New project → App, then New target → Audio Unit Extension) and then build and run it with either Logic Pro X or Garage Band, the plugin only functions when the track it's inserted on is selected.
If any other track is selected, breakpoints in eg. the process or handleMIDIEvent overriden functions never get triggered. (plus the unselected tracks start to output a constant, short period glitch noise if they were actually outputting sound before the selected track changed)
Any idea why this happens? I would suspect a fault in XCode or the DAW's part, but I have seen other macOS AUv3 plugins (a still extremely rare breed, unfortunately) work just fine, so I know it's definitely possible.
After much fiddling, I finally found the problem. (I REALLY wish there was more knowledge widely available online on AUv3...)
It seems that both Logic Pro X and Garage Band on each render cycle ask the plugin process for blocks of different lengths depending on whether the plugin is in the selected track or not. If the track is selected, the requested block will be the length set in the DAW's settings (I/O Buffer Size), presumably for highest priority rendering? Unselected tracks are asked for 1024 frames (the longest Logic's buffers can go, it seems), regardless of I/O Buffer Size setting.
1024 frames is longer than the
AUAudioFrameCount maxFramesToRender = 512;
the Audio Unit Extension template stubs in DSPKernel.hpp, thus making it so that rendering fails only when on an unselected track. (the short period glitch noise I mentioned appears to be whatever values were left in the output buffer since the last playback being re-output once every 1024 frames)
Setting maxFramesToRender = 1024; fixes that problem.
And now for a heavily opinionated rant:
I can't help but feel this default maxFramesToRender value is setting newbies (like me) up for failure, since 1) it's never mentioned in the official tutorials or documentation (AFAIK) 2) it doesn't play nice with Apple's own DAWs, presumably the most obvious places to test 3) it initially works, but only up until you try to play two tracks at once, at which point you probably already have a lot of code down and are all the more susceptible to confusion.
But oh well, I guess it is what it is.

Control Chromecast buffering at start

Is there a way to control the amount of buffering CC devices do before they start playback?
My sender apps sends real time audio flac and CC waits +10s before starting to play. I've built a customer receiver and tried to change the autoPauseDuration and autoResumeDuration but it does not seem to matter. I assume it's only used when an underflow event happens, but not at startup.
I realize that forcing a start with low buffering level might endup in underflow, but that's a "risk" that is much better than always waiting such a long time before playback starts. And if it happens, the autoPause/Resume hysteresis would allow a larger re-buffering to take place then.
If you are using the Media Player Library, take a look at player.getBufferDuration. The docs cover more details about how you can customize the player behavior: https://developers.google.com/cast/docs/player#frequently-asked-questions
Finally, it turned to be a problem with the way to send audio to the default receiver. I was streaming flac, and as it is a streamable format, I did not include any header (you might be able to start anywhere in the stream, it's just a matter of finding the synchro). But the flac decoder in the CC does not like that and was talking 10+ second to start. As soon as I've added a STREAMINFO header, problem went away

Outputting Sound to Multiple Audio Devices Simultaneously

OK, the first issue. I am trying to write a virtual soundboard that will output to multiple devices at once. I would prefer OpenAL for this, but if I have to switch over to MS libs (I'm writing this initially on Windows 7) I will.
Anyway, the idea is that you have a bunch of sound files loaded up and ready to play. You're on Skype, and someone fails in a major way, so you hit the play button on the Price is Right fail ditty. Both you and your friends hear this sound at the same time, and have a good laugh about it.
I've gotten OAL to the point where I can play on the default device, and selecting a device at this point seems rather trivial. However, from what I understand, each OAL device needs its context to be current in order for the buffer to populate/propagate properly. Which means, in a standard program, the sound would play on one device, and then the device would be switched and the sound buffered then played on the second device.
Is this possible at all, with any audio library? Would threads be involved, and would those be safe?
Then, the next problem is, in order for it to integrate seamlessly with end-user setups, it would need to be able to either output to the default recording device, or intercept the recording device, mix it with the sound, and output it as another playback device. Is either of these possible, and if both are, which is more feasible? I think it would be preferable to be able to output to the recording device itself, as then the program wouldn't have to be running in order to have the microphone still work for calls.
If I understood well there are two questions here, mainly.
Is it possible to play a sound on two or more audio output devices simultaneously, and how to achieve this?
Is it possible to loop back data through a audio input (recording) device so that is is played on the respective monitor i.e for example sent through the audio stream of Skype to your partner, in your respective case.
Answer to 1: This is absolutely feasable, all independent audio outputs of your system can play sounds simultaneously. For example some professional audio interfaces (for music production) have 8, 16, 64 independent outputs of which all can be played sound simultaneously. That means that each output device maintains its own buffer that it consumes independently (apart from concurrency on eventual shared memory to feed the buffer).
How?
Most audio frameworks / systems provide functions to get a "device handle" which will need you to pass a callback for feeding the buffer with samples (so does Open AL for example). This will be called independently and asynchroneously by the framework / system (ultimately the audio device driver(s)).
Since this all works asynchroneously you dont necessarily need multi-threading here. All you need to do in principle is maintaining two (or more) audio output device handles, each with a seperate buffer consuming callback, to feed the two (or more) seperate devices.
Note You can also play several sounds on one single device. Most devices / systems allow this kind of "resources sharing". Actually, that is one purpose for which sound cards are actually made for. To mix together all the sounds produced by the various programs (and hence take off that heavy burden from the CPU). When you use one (physical) device to play several sounds, the concept is the same as with multiple device. For each sound you get a logical device handle. Only that those handle refers to several "channels" of one physical device.
What should you use?
Open AL seems a little like using heavy artillery for this simple task I would say (since you dont want that much portability, and probably dont plan to implement your own codec and effects ;) )
I would recommend you to use Qt here. It is highly portable (Win/Mac/Linux) and it has a very handy class that will do the job for you: http://qt-project.org/doc/qt-5.0/qtmultimedia/qaudiooutput.html
Check the example in the documentation to see how to play a WAV file, with a couple of lines of code. To play several WAV files simultaneously you simply have to open several QAudioOutput (basically put the code from the example in a function and call it as often as you want). Note that you have to close / stop the QAudioOutput in order for the sound to stop playing.
Answer to 2: What you want to do is called a loopback. Only a very limited number of sound cards i.e audio devices provide a so called loopback input device, which would permit for recording what is currently output by the main output mix of the soundcard for example. However, even this kind of device provided, it will not permit you to loop back anything into the microphone input device. The microphone input device only takes data from the microphone D/A converter. This is deep in the H/W, you can not mix in anything on your level there.
This said, it will be very very hard (IMHO practicably impossible) to have Skype send your sound with a standard setup to your conversation partner. Only thing I can think of would be having an audio device with loopback capabilities (or simply have a physical cable connection a possible monitor line out to any recording line in), and have then Skype set up to use this looped back device as an input. However, Skype will not pick up from your microphone anymore, hence, you wont have a conversation ;)
Note: When saying "simultaneous" playback here, we are talking about synchronizing the playback of two sounds as concerned by real-time perception (in the range of 10-20ms). We are not looking at actual synchronization on a sample level, and the related clock jitter and phase shifting issues that come into play when sending sound onto two physical devices with two independent (free running) clocks. Thus, when the application demands in phase signal generation on independent devices, clock recovery mechanisms are necessary, which may be provided by the drivers or OS.
Note: Virtual audio device software such as Virtual Audio Cable will provide virtual devices to achieve loopback functionnality in Windows. Frameworks such as Jack Audio may achieve the same in UX environment.
There is a very easy way to output audio on two devices at the same time:
For Realtek devices you can use the Audio-mixer "trick" (but this will give you a delay / echo);
For everything else (and without echo) you can use Voicemeeter (which is totaly free).
I have explained BOTH solutions in this video: https://youtu.be/lpvae_2WOSQ
Best Regards

Flex 4 > spark.components.VideoPlayer > How to switch bit rate?

The VideoPlayer (possibly VideoDisplay also) component is capable of somehow automatically picking the best quality video on the list it's given. An example is here:
http://help.adobe.com/en_US/FlashPlatform/beta/reference/actionscript/3/spark/components/mediaClasses/DynamicStreamingVideoItem.html#includeExamplesSummary
I cannot find the answers to below questions.
Assuming that the server that streams recorded videos is capable of switching across same videos with different bit rates and streaming them from any point within their timelines:
Is the bandwidth test/calculation within this component only done before the video starts playing, at which point it picks the best video source and never uses the other ones? Or, does it continuously or periodically execute its bandwidth tests and does it accordingly switch between video sources during the playback?
Does it support setting the video source through code and can its automatic switching between video sources be turned off (in case I want to provide this functionality to the user in the form of some button/dropdown or similar)? I know that the preferred video source can be set, but this only means that that video source will be tested/attempted first.
What other media servers can be used with this component, besides the one provided by Adobe, to achieve automated and manual switching between different quality of same video?
Obviously, I'd like to create a player that is smart enough to automatically switch between different quality of videos, and that will support manual instructions related to which source to play - both without interrupting the playback, or at least without restarting it (minor interruptions acceptable). Also, the playback needs to be able to start at any given point within the video, after enough data has been buffered (of course), but most importantly, I want to be able to start the playback beyond what's buffered. A note or two about fast-forwarding is not going to hurt, if anyone knows anything.
Thank you for your time.

Looking for an OSX application that can do image processing using a webcam

I'm looking for an OSX (or Linux?) application that can recieve data from a webcam/video-input and let you do some image processing on the pixels in something similar to c or python or perl, not that bothered about the processing language.
I was considering throwing one together but figured I'd try and find one that exists already first before I start re-inventing the wheel.
Wanting to do some experiments with object detection and reading of dials and numbers.
If you're willing to do a little coding, you want to take a look at QTKit, the QuickTime framework for Cocoa. QTKit will let you easity set up an input source from the webcam (intro here). You can also apply Core Image filters to the stream (demo code here). If you want to use OpenGL to render or apply filters to the movie, check out Core Video (examples here).
Using theMyMovieFilter demo should get you up and running very quickly.
Found a cross platform tool called 'Processing', actually ran the windows version to avoid further complications getting the webcams to work.
Had to install quick time, and something called gVid to get it to work but after the initial hurdle coding seems like C; (I think it gets "compiled" into Java), and it runs quite fast; even scanning pixels from the webcam in real time.
Still to get it working on OSX.
Depending on what processing you want to do (i.e. if it's a filter that's available in Apple's Core Image filter library), the built-in Photo Booth app may be all you need. There's a comercial set of add-on filters available from the Apple store as well (http://www.apple.com/downloads/macosx/imaging_3d/composerfxeffectsforphotobooth.html).

Resources