For Dual Playback Demo page
<video class="primary"></video>
<video class="secondary" muted autoplay loop src="//storage.googleapis.com/ytlr-cert.appspot.com/test-materials/media/manual/dual_video/secondary-video.mp4"
maximum-capabilities="width=432; height=240; framerate=15;"></video>
How to distinguish between primary video and secondary video while StarboardPlayer is being created?
In StarboardPlayer constructor
StarboardPlayer::StarboardPlayer(
... [parameters skipped]
const std::string& max_video_capabilities)
max_video_capabilities string parameter is always empty on both primary and secondary, the page mentioned above doesn't use HTMLVideoElement.setMaxVideoCapabilities() to set max_video_capabilities parameter and following attribute in secondary video also not handled by HTMLMediaElement
maximum-capabilities="width=432; height=240; framerate=15;"
For resource limited system, minor resource allocated for secondary video during playback construction. For example, our system has only one resource which can be used in "punch out" output mode for primary video, so "decode to texture" output mode used in secondary video. After playback constructed, it seems not possible to change output mode dynamically? And memory resource also limited for secondary video.
So it's very important to distinguish between primary video and secondary video on resource limited embedded system, but it seems not possible in starboard layer and I don't want to modify code outside of starboard.
Any suggestions?
I noticed following code was added to dual_video.js. Now starboard layer can use max_video_capabilities string to determine whether it is secondary video or not.
Thank you :)
var secondaryVideo = document.querySelector('video.secondary');
secondaryVideo.setMaxVideoCapabilities('width=432; height=240; framerate=15;');
secondaryVideo.loop = true;
secondaryVideo.autoplay = true;
secondaryVideo.muted = true;
secondaryVideo.src = 'https://storage.googleapis.com/ytlr-cert.appspot.com/test-materials/media/manual/dual_video/secondary-video.mp4';
Related
I'm trying to log the time the GPU takes to render a frame. To do this I found that Unity implemented a struct FrameTiming, and a class named FrameTimingManager
The FrameTiming struct has a property gpuFrameTime which sounds like exactly what I need, however the value is never set, and the documentation on it doesn't provide much help either
public double gpuFrameTime;
Description
The GPU time for a given frame, in ms.
Looking further I found the FrameTimingManager class which contains a static method for GetGpuTimerFrequency(), which has the not so helpful documentation stating only:
Returns ulong GPU timer frequency for current platform.
Description
This returns the frequency of GPU timer on the current platform, used to interpret timing results. If the platform does not support returning this value it will return 0.
Calling this method in an update loop only ever yields 0 (on both Window 10 running Unity 2019.3 and Android phone running Android 10).
private void OnEnable()
{
frameTiming = new FrameTiming();
}
private void Update()
{
FrameTimingManager.CaptureFrameTimings();
var result = FrameTimingManager.GetGpuTimerFrequency();
Debug.LogFormat("result: {0}", result); //logs 0
var gpuFrameTime = frameTiming.gpuFrameTime;
Debug.LogFormat("gpuFrameTime: {0}", gpuFrameTime); //logs 0
}
So what's the deal here, am I using the FrameTimeManager incorrectly, or are Windows and Android not supported (Unity mentions in the docs that not all platforms are supported, but nowhere do they give a list of supported devices..)?
While grabbing documentation links for the question I stumbled across some forum posts that shed light on the issue, so leaving it here for future reference.
The FrameTimingManager is indeed not supported for Windows, and only has limited support for Android devices, more specifically only for Android Vulkan devices. As explained by jwtan_Unity on the forums here (emphasis mine):
FrameTimingManager was introduced to support Dynamic Resolution. Thus, it is only supported on platforms that support Dynamic Resolution. These platforms are currently Xbox One, PS4, Nintendo Switch, iOS, macOS and tvOS (Metal only), Android (Vulkan only), Windows Standalone and UWP (DirectX 12 only).
Now to be able to use the FrameTimingManager.GetGpuTimerFrequency() we need to do something else first. We need to take a snapshot of the current timings using FrameTimingManager.CaptureFrameTimings first (this needs to be done every frame). From the docs:
This function triggers the FrameTimingManager to capture a snapshot of FrameTiming's data, that can then be accessed by the user.
The FrameTimingManager tries to capture as many frames as the platform allows but will only capture complete timings from finished and valid frames so the number of frames it captures may vary. This will also capture platform specific extended frame timing data if the platform supports more in depth data specifically available to it.
As explained by Timothyh_Unity on the forums hereenter link description here
CaptureFrameTimings() - This should be called once per frame(presuming you want timing data that frame). Basically this function captures a user facing collection of timing data.
So the total code to get the GPU frequency (on a supported device) would be
private void Update()
{
FrameTimingManager.CaptureFrameTimings();
var result = FrameTimingManager.GetGpuTimerFrequency();
Debug.LogFormat("result: {0}", result);
}
Note that all FrameTimingManager methods are static, and do not require you to instantiate a manager first
Why none of this is properly documented by Unity beats me...
In an app, I'm driving a laser projection device using a connected USB audio interface on macOS.
The laser device takes analog audio as an input.
As a safety feature, it would be great if I could make the audio output from my app the exclusive output, because any other audio from other apps or from the OS itself which is routed to the USB audio interface is mixed with my laser control audio, is unwanted and a potential safety hazard.
Is it possible on macOS to make my app's audio output exclusive? I know you can configure AVAudioSession on iOS to achieve this (somewhat - you can duck other apps' audio, but notification sounds will in turn duck your app), but is something like this possible on the Mac? It does not need to be AppStore compatible.
Yes, you can request that CoreAudio gives you exclusive access to an audio output device. This is called hogging the device. If you hogged all of the devices, no other application (including the system) would be able to emit any sound.
Something like this would do the trick for a single device:
AudioObjectPropertyAddress HOG_MODE_PROPERTY = { kAudioDevicePropertyHogMode, kAudioObjectPropertyScopeGlobal, kAudioObjectPropertyElementMaster };
AudioDeviceID deviceId = // your audio device ID
pid_t hoggingProcess = -1; // -1 means attempt to acquire exclusive access
UInt32 size = sizeof(pid_t);
AudioObjectSetPropertyData(deviceId, &HOG_MODE_PROPERTY, 0, NULL, size, &hoggingProcess);
assert(hoggingProcess == getpid()); // check that you have exclusive access
Hog mode works by setting an AudioObject property called kAudioDevicePropertyHogMode. The value of the property is -1 if the device is not hogged. If it is hogged the value is the process id of the hogging process.
If you jump to definition on kAudioDevicePropertyHogMode in Xcode you can read the header doc for the hog mode property. That is the best way to learn about how this property (and pretty much anything and everything else in CoreAudio) works.
For completeness, here's the header doc:
A pid_t indicating the process that currently owns exclusive access to the
AudioDevice or a value of -1 indicating that the device is currently
available to all processes. If the AudioDevice is in a non-mixable mode,
the HAL will automatically take hog mode on behalf of the first process to
start an IOProc.
Note that when setting this property, the value passed in is ignored. If
another process owns exclusive access, that remains unchanged. If the
current process owns exclusive access, it is released and made available to
all processes again. If no process has exclusive access (meaning the current
value is -1), this process gains ownership of exclusive access. On return,
the pid_t pointed to by inPropertyData will contain the new value of the
property.
I'm using the C++ code given here. But the shared speech recognition used here runs its own commands such as move,minimize,delete. I need to create this without invoking the MS speech recognition program.
hr = cpEngine.CoCreateInstance(CLSID_SpSharedRecognizer);
this line above creates the shared instance.
I tried to use CLSID_SpInprocRecognizer instead but can not get it right. I'm new to this.
Is there a way to do this?
I met the same issue here, and spent lot of time trying to find an answer. Luckily, I've got the solution by following the steps:
Do use the in-process recognizer, if you want to get rid of the MS speech recognition program
hr = cpRecognizer.CoCreateInstance(CLSID_SpInprocRecognizer);
2.In-process recognizer doesn't have default input sources or recognition engines set up, and you need to set them to get the in-process recognizer to listen.
CComPtr<ISpObjectToken> cpObjectToken;
CComPtr<ISpAudio> cpAudio;
// Get the default audio input token.
hr = SpGetDefaultTokenFromCategoryId(SPCAT_AUDIOIN, &cpObjectToken);
// Set the audio input to our token.
hr = cpRecognizer->SetInput(cpObjectToken, TRUE);
// Set up the inproc recognizer audio input with an audio input object.
// Create the default audio input object.
hr = SpCreateDefaultObjectFromCategoryId(SPCAT_AUDIOIN, &cpAudio);
// Set the audio input to our object.
hr = cpRecognizer->SetInput(cpAudio, TRUE);
3.specifies the particular speech recognition engine to be used. If not specified, it will use the default one. If it's not called, it still use the default one(I commend out this line, still works fine).
hr = cpRecognizer->SetRecognizer(NULL);
That's it! It opens a default U.S. English recognition engine, and picks up my command pretty quick.
reference:
http://stackoverflow.com/questions/18448394/inproc-speech-recognition-engine-in-python
http://msdn.microsoft.com/en-us/library/ms718864%28v=vs.85%29.aspx
http://msdn.microsoft.com/en-us/library/ms718866%28v=vs.85%29.aspx
I'm working on a custom video transform filter derived from CTransformFilter. It doesn't do anything unusual in DirectShow terms such as extra internal buffering of media samples, queueing of output samples or dynamic format changes.
Graphs in graphedit containing two instance of my filters connected end to end (output of first connected to input of the second) hang when play is pressed. The graph definitely is not hanging within the ::Transform method override. The second filter instance is not connected directly to a video renderer.
The problem doesn't happen if a colour converter is inserted between the two filters. If I increase the number of buffers requested (ALLOCATOR_PROPERTIES::cBuffers) from 1 to 3 then the problem goes away. The original DecideBufferSize override is below and is similar to lots of other sample DirectShow filter code.
What is a robust policy for setting the number of requested buffers in a DirectShow filter (transform or otherwise)? Is code that requests one buffer out of date for modern requirements? Is my problem too few buffers or is increasing the number of buffers masking a different problem?
HRESULT MyFilter::DecideBufferSize(IMemAllocator *pAlloc, ALLOCATOR_PROPERTIES *pProp)
{
AM_MEDIA_TYPE mt;
HRESULT hr = m_pOutput->ConnectionMediaType(&mt);
if (FAILED(hr)) {
return hr;
}
BITMAPINFOHEADER * const pbmi = GetBitmapInfoHeader(mt);
pProp->cbBuffer = DIBSIZE(*pbmi);
if (pProp->cbAlign == 0) {
pProp->cbAlign = 1;
}
if (pProp->cBuffers == 0) {
pProp->cBuffers = 3;
}
// Release the format block.
FreeMediaType(mt);
// Set allocator properties.
ALLOCATOR_PROPERTIES Actual;
hr = pAlloc->SetProperties(pProp, &Actual);
if (FAILED(hr)) {
return hr;
}
// Even when it succeeds, check the actual result.
if (pProp->cbBuffer > Actual.cbBuffer) {
return E_FAIL;
}
return S_OK;
}
There is no specific policy on amount of buffers, though you should definitely be aware that fixed number of buffers is the method to control sample rate. When all buffers are in use, a request for another buffer will block execution until such buffer is available.
That is, if your code is holding buffer references for certain purpose, you should allocate the respective amount so that you don't lock yourself. E.g. you hold last media sample reference internally, e.g. to be able to re-send it, and you still want to be able to deliver other media samples, so you need at least two buffers on the allocator.
Output pin is typically responsible to choose and set up the allocator, and input might might need to check and update properties if/when it is notified which allocator is to be used. On inplace transformation filters when you share the allocators, you might want an additional check in order to make sure the requirements are met.
DMO Wrapper Filter uses (at least sometimes) allocators with one buffer only and is still in good standing
with audio you normally have more buffers because you queue data for playback
if you have a reference leak on your code and you don't release media sample pointers, then your streaming might lock dead because of this
I want to get the adpater RAM or graphics RAM which you can see in Display settings or Device manager using API. I am in C++ application.
I have tried seraching on net and as per my RnD I have come to conclusion that we can get the graphics memory info from
1. DirectX SDK structure called DXGI_ADAPTER_DESC. But what if I dont want to use DirectX API.
2. Win32_videocontroller : But this class does not always give you adapterRAM info if availability of video controller is offline. I have checked it on vista.
Is there any other way to get the graphics RAM?
There is NO way to directly get graphics RAM on windows, windows prevents you doing this as it maintains control over what is displayed.
You CAN, however, create a DirectX device. Get the back buffer surface and then lock it. After locking you can fill it with whatever you want and then unlock and call present. This is slow, though, as you have to copy the video memory back across the bus into main memory. Some cards also use "swizzled" formats that it has to un-swizzle as it copies. This adds further time to doing it and some cards will even ban you from doing it.
In general you want to avoid directly accessing the video card and letting windows/DirectX do the drawing for you. Under D3D1x Im' pretty sure you can do it via an IDXGIOutput though. It really is something to try and avoid though ...
You can write to a linear array via standard win32 (This example assumes C) but its quite involved.
First you need the linear array.
unsigned int* pBits = malloc( width * height );
Then you need to create a bitmap and select it to the DC.
HBITMAP hBitmap = ::CreateBitmap( width, height, 1, 32, NULL );
SelectObject( hDC, (HGDIOBJ)hBitmap );
You can then fill the pBits array as you please. When you've finished you can then set the bitmap's bits.
::SetBitmapBits( hBitmap, width * height * 4, (void*)pBits )
When you've finished using your bitmap don't forget to delete it (Using DeleteObject) AND free your linear array!
Edit: There is only one way to reliably get the video ram and that is to go through the DX Diag interfaces. Have a look at IDxDiagProvider and IDxDiagContainer in the DX SDK.
Win32_videocontroller is your best course to get the amount of gfx memory. That's how its done in Doom3 source.
You say "..availability of video controller is offline. I have checked it on vista." Under what circumstances would the video controller be offline?
Incidentally, you can find the Doom3 source here. The function you're looking for is called Sys_GetVideoRam and it's in a file called win_shared.cpp, although if you do a solution wide search it'll turn it up for you.
User mode threads cannot access memory regions and I/O mapped from hardware devices, including the framebuffer. Anyway, what you would want to do that? Suppose the case you can access the framebuffer directly: now you must handle a LOT of possible pixel formats in the framebuffer. You can assume a 32-bit RGBA or ARGB organization. There is the possibility of 15/16/24-bit displays (RGBA555, RGBA5551, RGBA4444, RGBA565, RGBA888...). That's if you don't want to also support the video-surface formats (overlays) such as YUV-based.
So let the display driver and/or the subjacent APIs to do that effort.
If you want to write to a display surface (which not equals exactly to framebuffer memory, altough it's conceptually almost the same) there are a lot of options. DX, Win32, or you may try the SDL library (libsdl).