A few years ago, DirectShow was around and let you manage video on DirectDraw surfaces. But since then I think both technologies have been replaced. What's currently the best solution to let you make a Windows app which can let you composite/blend/mix videos/music together? Does one still need to go the DirectX route with surfaces/textures, or is functionality found in the core Windows APIs?
Examples might be to overlay an image on a playing video, overlay two videos on top of each other with a transition effect, etc.
Apart from core technologies to handle video/audio, are their good 3rd-party libraries? Or maybe the core APIs have enough functionality on their own?
If you're talking managed code?
Microsoft.DirectX.AudioVideoPlayback
Short tutorial here:
http://forum.codecall.net/csharp-tutorials/20436-tutorial-playing-video-files-managed-directx.html
Related
Is there a good GUI for VR? Scaleform is no longer supported. Coherent seems like an option. I just don't know how to integrate it if it is not natively supported by the engine. IMGUI seems like it is limited in the look that can be created by it. Are there any others?
Have you looked at Qt? It's very flexible and works on a variety of different platforms. QtQuick/QML in particular is pretty nice for VR interfaces.
Here is a video of a talk from Qt World Summit 2017 that does a pretty good job of summarizing the current state of things: https://www.youtube.com/watch?v=RtcmVbicZWw (Basically, the maintainers of Qt are working on a lot of cool VR-related things). I don't work for them and have only starting using Qt recently, but from what I've seen so far it works very well for this purpose.
In particular, take a look at QQuickRenderControl (http://doc.qt.io/qt-5/qquickrendercontrol.html). This class allows full control over rendering of QML content to a framebuffer. Right now I'm using it in the creation of SteamVR overlays.
I've been recently investigating different multimedia frameworks for adding audio and video capabilities to my applications.
I've been looking at phonon, gstreamer, ffmpeg, libvlc/vlc.
However, I cannot find a good resource that answers some of my general questions.
Are these interchangeable?
Do they work in the same level?
Do you have any experience using some and can give feedback of why did you chose one over the other?
Thanks
Are these interchangeable?
generally not. Phonon is a high level api that wraps actual multimedia frameworks, which allows you to change the backend, but on the other hand limits what you can do.
Do they work in the same level?
no. some of the ones you mentioned are high level, some are low level.
Do you have any experience using some and can give feedback of why did you chose one over the other?
You should really tell what you want to do. Then people can advise what framework might be suitable. Lower level frameworks such as gstreamer cover quite a large variety of use cases.
There is a 'GStreamer SDK' for windows and OS/X which should get you started easily on those platforms (on Linux you can just install your distro's -dev packages). The SDK ships with snappy, which is a small media player using clutter, but you can easily build your own player using some other toolkit or API of course.
I have an old computer vision experiment that uses Video for Windows to grab frames from a camera connected to the PC. It's a hack, it uses VfW to create a preview window, then it does a GetDIBits from the window DC.
I'm finally ready to port this to DirectShow. My understanding was that I could grab frames from a video capture graph by using ISampleGrabber, but now I read that ISampleGrabber is deprecated.
What's the non-deprecated way to grab frames from a video feed? Do I have to implement my own DirectShow filter that does essentially what ISampleGrabber does?
DirectShow is not deprecated; just the DirectShow Editing Services. I would strongly recommend using DirectShow because of the much wider level of support, unless there are specific features of MF that are needed.
There's been no development of DES for some years, but the sample grabber is a widely-used filter that is somewhat independent of DES. I would be happy to recommend that you use it. If there is an issue in future versions of windows, it would not be more than a day or two's work to replace the filter.
G
I think Windows Media Foundation would be your best bet if you are only targeting Vista/Win7, otherwise you can still use DirectShow/SampleGrabber approach, I doubt it will be removed any time soon. Related question here.
I have this garden variety USB video camera, and it came with two mini-apps, one that just lets you see what the camera sees, and one that records to an .avi file.
But what's the API if I want to grab images from the camera in my own C program? I am making the assumptions that it's (1) possible and (2) desirable to make some call and have a 2D array of pixel information filled in.
What I really want to do is tinker with image processing algorithms, and for that I'd really like to get my code around some live data.
EDIT -
Having had a healthy exposure to Linux, I can grasp how (ideally/in theory) you could open() the device, use ioctl() to configure it, and read() the data. And I'm virtually certain that that's not how Windows is going to present the API. Not knowing what function names Windows might use for a video device API, or even if it has one, makes it difficult to look up, at least with the win32 api search capabilities that I have at my disposal.
You'll probably need the DirectShow API, provided that's how the camera operates. If the manufacturer created their own code path, you'll need their API.
Your first step, as pointed out by ChrisBD, is to check if Windows supports your device.
If that is the case you have three possible Windows APIs for capture:
DirectShow
VFW. Has more or less been replaced by DirectShow
MediaFoundation. Is the newest API that is intended to replace DirectShow. AFAIK not fully implemented yet and only available in Vista.
From the three DirectShow is the best choice. However, learning and using DirectShow is not a trivial task. An excellent example can be found here.
Another possibility is to use OpenCV. OpenCV is an image processing library, that you can also use for processing the images. OpenCV has an image capture API that provides a simpler abstraction and is easier to use than the Windows APIs.
The API is the way to go.
A good indication of whether the camera requires a bespoke one or not is to see if it is recognised by a PC without the manufacturer's applications installed. If windows has the drivers built in the you should be able to use the windows APIs to capture the images.
Alternatively if you know what compression codec has been used for the AVI file you could unpack it.
Ideally it would be good if you could capture the video in native (YUV, RGB15 or similar) format as then you can work on compression as well as manipulation.
I'm making an application where I will:
Record from the microphone and do some realtime processing on the input
Play an MP3 file (a regular song), but manipulating the output in realtime
Every now and then I'll need to play additional sounds over this song too, but I guess I can do that by simply adding the buffers.
In short, I need to have circular buffers for both recording and playing, and I need to be "feeding" the output buffer every 20 ms or so with the new data that is just about to be played.
I've been looking at DirectSound, and it doesn't seem to help a lot. The reading and writing to the output buffers seem very similar to Win32, the only place where it seems it'd help is in playing the "additional sounds" over the main song.
Should I use DirectSound, or should I go straight to raw Windows APIs?
Is DirectSound going to do anything for me?
Thanks in Advance!
The Directsound API's give you better realtime control. They are also the supported way to use sound in Windows. I was under the impression that the win32 api's were depracated, but I could be wrong on this.
This question is close to yours
https://stackoverflow.com/questions/314522/what-is-the-best-c-sound-api-for-windows
also
Is DirectSound the best audio abstraction layer for Windows?
last but not least, this is what microsoft has to say http://msdn.microsoft.com/en-us/library/dd370784(VS.85).aspx
Neither? :)
The story is that DirectSound is the replacement for waveOut, but DirectSound joined DirectInput as deprecated APIs in Vista and is replaced with WASAPI. DirectSound and waveOut are implemented on top of the User-Space WASAPI in Vista. On XP, waveOut and DirectSound feed to the same kernel level Mixer API.
To consolidate all of these interfaces take a look at something like OpenAL, it's a well supported audio standard along the same lines as OpenGL.
It sounds like you're going to be quite sensitive to latency. It might pay to look at ASIO
I found Harmony Central - Audio Programming. Also read w:DirectSound.
Windows Vista features a completely
re-written audio stack based on the
Universal Audio Architecture. Because
of the architectural changes in the
redesigned audio stack, a direct path
from DirectSound to the audio drivers
does not exist.
Because of Xbox 360 and Microsoft
Windows integration, Microsoft is
actively pushing developers to migrate
new applications to equivalent Xbox
audio APIs such as XAudio and XACT.
OpenAL looks promising.