I developed an application in C that can display 4 videos, and 1 sound file in the background.
The video uses the WMP object in the C++ class provided by Microsoft in the WMP SDK. The audio uses Windows' MCI (Media Control Interface) which is sent command strings.
To be able to play lots of different formats, I installed windows.7.codec.pack.
I experience a problem that when playing more than 3 media files (video or audio), the media stall. A video to be started constantly cycles from state 3 (Playing) to state 9 (Preparing new media) to state 10 (Ready to begin playing - without anything happening), and so on and so on. This is seen as a flicker of the video (state 3) followed by a few seconds of nothing (black, or desktop background, state 9). Once a video has started, it plays fine until the end.
Decreasing the number of media files to play to 2 videos and 1 audio lets it play fine; increasing to 3 video or more and 1 audio and this happens. Task Manager shows a CPU load of less than 25%, so CPU cannot be the problem.
It seems that MCI and WMP share stuff in the background because not only do videos stall, also audio stops without reporting errors (querying MCI returns that it is playing, but there is no sound).
I upgraded to windows.7.codec.pack.v4.2.6. This had a terrible performance
I reverted to windows.7.codec.pack.v4.1.6. This has a much better performance but still not perfect.
My question(s):
Is there any way I can configure Windows or the codec package to seamlessly handle 5 streams?
How can I report this problem to the developer?
Are there other codec packages that do not have this problem?
Any other suggestion?
System info: Intel i7-3520M X64 dual core at 2.9Ghz with 8GB physical memory and NVIDIA Quatro K1000M display adapter.
I think I found the solution.
After playing a video, I called the Player's Close() method. The documentation says:
The close method releases Windows Media Player resources.
Remarks
This method closes the current digital media file, not the Player itself.
Indeed I wanted to release the media file so it would no longer be busy in the file system. However, it seems that more resources were released than just the media file. As a consequence, for a next media file to be played, the player had to allocate resources again. It seems that turned out to be a bottle neck.
No longer calling Close() but just giving it the URL (filename) of the next media file to play now solved the problem. (I still have to give some retries sometimes but the general performance is now very well acceptable.)
The media file is released when the next media file starts playing.
Related
Trying to play a 3840x2160 video recorded by an iPhone 7 (#30fps), I get frequent pauses -- in the video, music keeps playing.
This happens both in firefox and when ffplay is invoked to play the file directly -- from command-line. The CPU is a dual E6700 #3.20GHz -- not super fast, but it should be able to play smoothly, shouldn't it? Video is Intel's "series 4" integrated chipset -- again, not a speed-daemon, but it should be adequate... Support for Intel's VA API is included.
I build ffmpeg-4.1 from source using FreeBSD port. As you can see, the port has a maddening amount of options -- including several different ones for the 264-codec.
Any suggestions for improving the decoding speed to the point, where it is watchable by a human? Thank you!
I notice that in the Microsoft SDK audio samples, both of the Render/Exclusive examples will, for example, stop iTunes playing through my headphones (wholly expected), but when the example program has completed, iTunes is left in a kind of zombie state that requires it to be closed and restarted in order to play anything.
Is this normal, or a fault of iTunes (my version: 11.1.3.8)... or is there something an application should do to announce to other applications that its exclusive use of the endpoint has finished?
I want to create a virtual audio device that gets audio data from the default output (which is an output IOAudioStream) and converts it to an input IOAudioStream.
I went through most of the examples I could find, however they only implement a feature to copy the output IOAudioStream to the input one at most. That means it only converts the audio to an input stream if the audio device is selected as output.
This should be possible, since ScreenFlow allows recording of computer audio by installing a kext that creates a virtual driver.
How can I access the audio data from the default output and send it to my virtual driver?
Take a look at the open-source WavTap, which is a simplified fork of the open-source SoundFlower virtual sound card driver. It is a .kext that I believe does substantially what you want.
For reference, here is how some popular commercial closed-source options work:
Rogue Amoeba's Audio Hijack Pro
-Captures system audio via code based off of the open-source SoundFlower .kext
-Captures an application's audio by substituting a "patch" framework for the normal CoreAudio.framework when launching the application
-Captures an already-running application's audio with the help of the haxie "Application Enhancer" (APE) from Unsanity
These features are branded as their "Instant On" feature (InstantOn.kext).
Ambrosia Software's WireTap Studio
-Captures system audio and application audio via an in-house developed .kext
Telestream's ScreenFlow
-Captures system audio via an in-house developed .kext. (Version 2.x uses varaudio.kext; Version 3.x uses TelestreamAudio.kext)
Macsome's Audio Recorder
-Unknown method
Araelium Group's Screenflick
-Captures system audio using the SoundFlower .kext
UPDATE #1
After reading the author's comments, it appears the underlying goal is to be able to capture the system sound without publishing the virtual audio driver as a device (that would appear in the System Preference's list) and without changing the current default output device (or at least the appearance that the device has changed).
SoundFlower: Adds a sound device to the list upon installation
WavTap: Adds a sound device to the list upon installation; auto-selects the device when the WavTap application is started; auto-deselects the device when the application is shutdown and reselects the previous device
Audio Hijack Pro: Adds a sound device only when audio capture of the default system sound is selected; removes the sound device when audio capture is no longer selected and reselects the previous device
WireTap Studio: Unknown
ScreenFlow: Captures the system sound without changing the current default output device and without publishing the virtual audio driver as a device
UPDATE #2
A quote from Jeff Moore, a CoreAudio Apple engineer, in reference to applications such as WireTap and Audio Hijack Pro:
"There are no APIs on the system that will give you the output of any specific app or the whole mix going to the hardware...[Capturing System Sound] isn't supported by the System and those folks had to be clever. There's nothing stopping you from doing the same thing except how willing you are to get your hands dirty.
The fact is, Mac OS X's audio system was designed first and foremost for performance. This lead us to a design where it is not easy to support the functionality you want without imposing performance penalties. So, we have opted for better performance at the cost of not being able to provide this feature."
If you want to read more on the subject, check out these threads on the CoreAudio API mailing list:
"WireTap, CoreAudio's API, and system capture, and kexts..."
"Another question on capturing audio played back by a software"
"Capturing currently played audio using CoreAudio on Mac"
"'audio hijack'"
"monitoring system audio output like wire tap"
"Capturing audio output to a file"
"Mirroring Audio Output"
"Recording system audio"
Relevant SO Questions:
Hide Audio device using codeless kext
So long story short, you're not likely to find examples from Apple that accomplish this, and you're not likely to find open source code that accomplishes this either, unless someone is feeling very generous. It appears to be too valuable of information.
After additional research, here are some theoretical techniques I came up with that might allow you to accomplish your goal:
Similar to Prosoft Engineering's Hear product, you could create a HAL plugin (user-mode virtual driver) rather than a .kext (kernel-mode virtual driver). Apple has a sample HAL plugin called "SampleHardwarePlugIn" and PulseAudio has one as well. However, with his method I don't think you get access to a pre-mixed system sound stream. You would have to gather up all streams from the various applications (which must use CoreAudio to play sound) and mix them together for pseudo system sound capture.
Create a virtual audio device that is hidden [1][2] from user interaction. When the user wishes to capture the default sound, programmatically create an aggregate device that includes your hidden virtual device and the current default sound device. Temporarily set this aggregate device as the default output. In this manner, you are able to both capture the default sound and hear it.
Side Note: If Mac OS X allows for a hidden device to also be set as the default output device, what would System Preferences show as the selected device? If it instead shows the secondary output device as selected, then you have the added allusion that nothing has changed.
A newer open-source virtual audio device that works with the latest versions of MacOS is BlackHole - it supports multiple audio channels and sampling rates.
It can be used as an audio sink and/or source. It's also handy as part of an aggregate audio device so audio can be heard and re-routed - e.g. using the MacOS Audio MIDI Setup app
I'm using ManagedMEdiaHelpers project as a base to a background audio streaming project.
The audio output is fine but sometimes is takes 1 to 6 seconds to start playing. During that time the device sends some strange noises similar to hiccups or scratching.
The mp3 I'm trying to stream have the following properies:
Bitrate: 320000
Sampling Rate: 44100
What are the possible causes to receive that kind of noises on the start of the stream when the rest of the mp3 plays just fine?
More info
I noticed that during the hiccups the fps count was below 20 so I tried to profile the application.
I got the following message during the hiccups period:
Waring :Very high CPU usage by system threads: System and other applications
are using 65,02% of the CPU. This CPU usage may be caused by other
tasks that are running on the system or they may be caused by system
operations that are triggered by a user application. Ensure that no
other tasks are running on the system during profiling.
It was an hardware limitation of HTC Radar.
I Just tried the same code on an Samsung OMNIA 7 and the stream is just perfect. Also there's no penalty on the fps count so I think that on this device there's no CPU hogging.
Strangely gsmarena says that both devices have the same CPU.
The software in question is a native C++/MFC application that receives a large amount of data over UDP and then processes the data for display, sound output, and writing to disk among other things. I first encountered the problem when the application's CHM help document was launched from its help menu and then I clicked around the help document while gathering data from the hardware. To replicate this, an AutoHotkey script was used to rapidly click around in the help document while the application was running. As soon as any sound occurred on the system, I started getting errors.
If I have the sound card completely disabled, everything processes fine with no errors, though sound output is obviously disabled. However, if I have sound playing (in this application, a different application or even just the beep from a message box) I get thousands of dropped packets (we know this because each packet is timestamped). As a second test, I didn't use my application at all and just used Wireshark to monitor incoming packets from the hardware. Sure enough, whenever a sound played in Windows, we had dropped packets. In fact, sound doesn't even have to be actively playing to cause the error. If I simply create a buffer (using DirectSound8) and never start playing, I still get these errors.
This occurs on multiple PCs with multiple combinations of network cards (both fiber optic and RJ45) and sound cards (both integrated and separate cards). I've also tried different driver versions for each NIC and sound card. All tests have been on Windows 7 32bit. Since my application uses DirectSound for audio, I've tried different CooperativeLevels (normal operation is DSSCL_PRIORITY) with no success.
At this point, I'm pretty convinced it has nothing to do with my application and was wondering if anyone had any idea what could be causing this problem before I started dealing with the hardware vendors and/or Microsoft.
It turns out that this behavior is by design. Windows Vista and later implemented something called the Multimedia Class Scheduler service (MMCSS) that is intended to make all multimedia playback as smooth as possible. Since multimedia playback relies on hardware interrupts to ensure smooth playback, any competing interrupts will cause problems. One of the major hardware interrupt sources is network traffic. Because of this, Microsoft decided to throttle the network traffic when a program was running under MMCSS.
I guess this was a big deal back in 2007 when Vista came out, but I missed it. There was an article by Mark Russinovich (thanks ypnos) describing MMCSS. It seems that the my entire problem boiled down to this:
Because the standard Ethernet frame
size is about 1500 bytes, a limit of
10,000 packets per second equals a
maximum throughput of roughly 15MB/s.
100Mb networks can handle at most
12MB/s, so if your system is on a
100Mb network, you typically won’t see
any slowdown. However, if you have a
1Gb network infrastructure and both
the sending system and your Vista
receiving system have 1Gb network
adapters, you’ll see throughput drop
to roughly 15%. Further, there’s an
unfortunate bug in the NDIS throttling
code that magnifies throttling if you
have multiple NICs. If you have a
system with both wireless and wired
adapters, for instance, NDIS will
process at most 8000 packets per
second, and with three adapters it
will process a maximum of 6000 packets
per second. 6000 packets per second
equals 9MB/s, a limit that’s visible
even on 100Mb networks.
I haven't verified that the multiple adapter bug still exists in Windows 7 or Vista SP1, but it is something to look for if you are running into problems.
From the comments on Russinovich's post, I found that Vista SP1 introduced some registry settings that allowed one to adjust how MMCSS affects Windows. Specifically the NetworkThrottlingIndex key.
The solution to my issue was to completely disable network throttling by setting the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile\NetworkThrottlingIndex key to 0xFFFFFFFF and then rebooting. This completely disables the network throttling portion of MMCSS. I had tried simply upping the value to 70, but it didn't stop causing errors until I completely disabled it.
Thus far I have not seen any adverse effects on other multimedia applications (nor the video capture and audio output portions of my own application) from this change. I will report back here if that changes.
It is known that Microsoft built some weird anti-feature into the Windows Vista kernel that will degrade I/O performance preventatively to make sure that multimedia applications (windows media player, directX) get 100% responsiveness. I don't know if that also means packet loss with UDP. Read this lame justification for the method: http://blogs.technet.com/b/markrussinovich/archive/2007/08/27/1833290.aspx
One of the comments there summarizes this quite well: "Seems to me Microsoft tried to 'fix' something that wasn't broken."