What is the preferred way of synchronizing with monitor refreshes, when vsync is not an option? We enable vsync, however, some users disable it in driver settings, and those override app preferences. We need reliable predictable frame lengths to simulate the world correctly, do some visual effects, and synchronize audio (more precisely, we need to estimate how long a frame is going to be on screen, and when it will be on screen).
Is there any way to force drivers to enable vsync despite what the user set in the driver? Or to ask Windows when a monitor rerfesh is going to happen? We have issues with manual sleeping when our frame boundaries line up closely to vblank. It causes occasional missed frames, and up to 1 extra frame of input latency.
We mainly use OpenGL, but Direct3D advice is also appreciated.
You should not build your application's timing on the basis of vsync and exact timings of frame presentation. Games don't do that these days and have not do so for quite some time. This is what allows them to keep a consistent speed even if they start dropping frames; because their timing, physics computations, AI, etc isn't based on when a frame gets displayed but instead on actual timing.
Game frame timings are typically sufficiently small (less than 50ms) that human beings cannot detect any audio/video synchronization issues. So if you want to display an image that should have a sound played alongside it, as long as the sound starts within about 30ms or so of the image, you're fine.
Oh and don't bother trying to switch to Vulkan/D3D12 to resolve this problem. They don't. Vulkan in particular decouples presentation from other tasks, making it basically impossible to know the exact time when an image starts appearing on the screen. You give Vulkan an image, and it presents it... at whatever is the next most opportune moment. You get some control over how that moment gets chosen, but even those choices can be restricted based on factors outside of your control.
Design your program to avoid the need for rigid vsync. Use internal timings instead.
Related
I have windowed WinApi/OpenGL app. Scene is drawn rarely (compared to games) in WM_PAINT, mostly triggered by user input - MW_MOUSEMOVE/clicks etc.
I noticed, that when there is no scene moving by user mouse (application "idle") and then some mouse action by user starts, the first frame is drawn with unpleasant delay - like 300 ms. Following frames are fast again.
I implemented 100 ms timer, which only does InvalidateRect, which is later followed by WM_PAINT/draw scene. This "fixed" the problem. But I don't like this solution.
I'd like know why is this happening and also some tips how to tackle it.
Does OpenGL render context save resources, when not used? Or could this be caused by some system behaviour, like processor underclocking/energy saving etc? (Although I noticed that processor runs underclocked even when app under "load")
This sounds like Windows virtual memory system at work. The sum of all the memory use of all active programs is usually greater than the amount of physical memory installed on your system. So windows swaps out idle processes to disc, according to whatever rules it follows, such as the relative priority of each process and the amount of time it is idle.
You are preventing the swap out (and delay) by artificially making the program active every 100ms.
If a swapped out process is reactivated, it takes a little time to retrieve the memory content from disc and restart the process.
Its unlikely that OpenGL is responsible for this delay.
You can improve the situation by starting your program with a higher priority.
https://superuser.com/questions/699651/start-process-in-high-priority
You can also use the virtuallock function to prevent Windows from swapping out part of the memory, but not advisable unless you REALLY know what you are doing!
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366895(v=vs.85).aspx
EDIT: You can improve things for sure by adding more memory and for sure 4GB sounds low for a modern PC, especially if you Chrome with multiple tabs open.
If you want to be scientific before spending any hard earned cash :-), then open Performance Manager and look at Cache Faults/Sec. This will show the swap activity on your machine. (I have 16GB on my PC so this number is very low mostly). To make sure you learn, I would check Cache Faults/Sec before and after the memory upgrade - so you can quantify the difference!
Finally, there is nothing wrong with the solution you found already - to kick start the graphic app every 100ms or so.....
Problem was in NVidia driver global 3d setting -"Power management mode".
Options "Optimal Power" and "Adaptive" save power and cause the problem.
Only "Prefer Maximum Performance" does the right thing.
Apple's I/O kit (via IODisplaySetFloatParameter) allows you to set the display brightness within a given range. However, I remember my prior laptops being significantly dimmer at the lowest setting.
Various screen dimming utilities alter the Gamma settings, and this brings the display down even further. However, the qualitative difference in the change and how such these utilities use RGB tables leads me to suspect that the Gamma setting ONLY alters the color tables, not the LED backlight.
Does anyone know of private API's (or how I would find them) that let me set the display to something lower than what IODisplaySetFloatParameter allows?
The hardware for this kind of thing tends to use PWM (Pulse Width Modulation), since LEDs are not inherently dimmable; that is, the hardware will switch the LEDs off and on very quickly, ensuring that when it is set to maximum brightness the LEDs are on 100% of the time, while at minimum brightness the LED will actually be switched off for most of each cycle.
That leads to the following observation: there is no particular reason you can't implement your own PWM in software, toggling the backlight on and off and controlling the proportion of the time it spends in each state. The downside is that you want to do this toggling fast to avoid it looking like flickering, and that will burn some CPU. You might want to investigate whether it's best to let the hardware PWM run as well as yours (since then yours can run somewhat slower), by setting the display brightness at a value other than 100% for the “on” part of the cycle.
Anyway, just an idea.
I am writing a client/server app in that server send live audio data that capture audio samples that captured from some external device( mic. for example ) and send it to the client. Then client want to play those samples. My app will run on local network so I have no problem with bandwidth( My sound is 8k, 8bit stereo while my net card 1000Mb ). In client I buffer the data for a small time and then start playback. and as data arrive from server I send them to sound card. This seems to work fine but there is a problem:
when my buffer in the client side finished, I will experience gaps in played sound.
I consider this is because of the difference in sampling time of the server and the client, it means that 8K on server is not same as 8K on client.
I can solve this with pausing client's playback and buffer again, but my boss doesn't accept it, since I have proper bandwidth and I should be able to play sound with no gap or pause.
So I decided to dynamically change playback speed in the client but I don't know how.
I am programming in Windows( native ) and I currently use waveOutXXX to play the sound. I can use any other native library( DirectX/DirectSound, Jack or ... ) but they should provide a smooth playback in the client.
I have programmed with waveOutXXX many times without any problem and I know it good but I can't solve my problem of dynamic resampling
I would suggest that your problem isn't likely due to mis-matched sample rates, but something to do with your buffering. You should be continuously dumping data to the sound card, and continuously filling your buffer. Use a reasonable buffer size... 300ms should be enough for most applications.
Now, over long periods of time, it is possible for the clock on the recording side and the clock on the playback side to drift apart enough that the 300ms buffer is no longer sufficient. I would suggest that rather than resampling at such a small difference, which could introduce artifacts, simply add samples at the encoding end. You still record at 8kHz, but you might add a sample or two every second, to make that 8.001kHz or so. Simply doubling one of the existing samples for this (or even a simple average between one sample and the next) will not be audible. Adjust this as necessary for your application.
I had a similar problem in an application I worked on. It did not involve network, but it did involve source data being captured in real-time at a certain fixed sampling rate, a large amount of signal processing, and finally output to the sound card at a fixed rate. Like you, I had gaps in the playback at buffer boundaries.
It seemed to me like the problem was that the processing being done caused audio data to make it to the sound card in a very jerky manner. That is, it would get a large chunk, then it would be a long time before it got another chunk. The overall throughput was correct, but this latency caused the sound card to often be starved for data. I suppose you may have the same situation with the network piece in your system.
The way I solved it was to first make the audio buffer longer. Then, every time a new chunk of audio was received, I checked how full the buffer was. If it was less than 20% full, I would write some silence to make it around 60% full.
You may think that this goes against reducing the gaps in playback since it is actually adding a gap, but it actually helps. The problem that I was having was that even though I had a significantly large audio buffer, I was always right at the verge of it being empty. With the other latencies in the system, this resulted in playback gaps on almost every buffer.
Writing the silence when the buffer started to get empty, but before it actually did, ensured that the buffer always had some data to spare if the processing fell behind a little. Also, just a single small gap in playback is very hard to notice compared to many periodic gaps.
I don't know if this will work for you, but it should be easy to implement and try out.
The setup
The game in question is using CoreAudio and single AudioGraph to play sounds.
The graph looks like this:
input callbacks -> 3DMixer -> DefaultOutputDevice
3DMixer's BusCount is set to 50 sounds max.
All samples are converted to default output device's stream format before being fed to input callbacks. Unused callbacks aren't set (NULL). Most sounds are 3D, so azimuth, pan, distance and gain are usually set for each mixer input, not left alone. They're checked to make sure only valid values are set. Mixer input's playback rate is also sometimes modified slightly to simulate pitch, but for most sounds it's kept at default setting.
The problem
Let's say I run the game and start a level populated with many sounds, lot of action.
I'm running HALLab -> IO Cycle Telemetry window to see how much time it takes to process each sound cycle - it doesn't take any more than 4ms out of over 10 available ms in each cycle, and I can't spot a single peak that would make it go over alloted time.
At some point when playing the game, when many sounds are playing at the same time (less than 50, but not less than 20), I hear a poof, and from then on only silence. No Mac sounds can be generated from any application on Mac. The IO Telemetry window shows my audio ticks still running, still taking time, still providing samples to output device.
This state persists even if there are less, then no sounds playing in my game.
Even if I quit the game entirely, Mac sounds generated by other applications don't come back.
Putting Mac to sleep and waking it up doesn't help anything either.
Only rebooting it fully results in sounds coming back. After it's back, first few sounds have crackling in them.
What can I do to avoid the problem? The game is big, complicated, and I can't modify what's playing - but it doesn't seem to overload the IO thread, so I'm not sure if I should. The problem can't be caused by any specific sound data, because all sound samples are played many times before the problem occurs. I'd think any sound going to physical speakers would be screened to avoid overloading them physically, and the sound doesn't have to be loud at all to cause the bug.
I'm out of ideas.
I need to write a screencast, and need to detect when window content has changed, even only text was selected. This window is third party control.
Ther're several methods.
(1) Screen polling.
You can poll the screen (that is, create a DIB, each time period to BitBlt from screen to it), and then send it as-is
Pros:
Very simple to implement
Cons:
High CPU load. Polling the entire screen number of times per second is very heavy (a lot of data should be transferred). Hence it'll be heavy and slow.
High network bandwidth
(2) Same as above, except now you do some analyzing of the polled screen to see the difference. Then you may send only the differences (and obviously don't send anything if no changes), plus you may optionally compress the differences stream.
Pros:
Still not too complex to implement
Significantly lower network bandwidth
Cons:
Even higher CPU usage.
(3) Same as above, except that you don't poll the screen constantly. Instead you do some hooking for your control (like spying for Windows messages that the control receives). Then you try learn when your control is supposed to redraw itself, and do the screen polling only in those scenarios.
Pros:
Significantly lower CPU usage
Still acceptable network bandwidth
Cons:
Implementation becomes complicated. Things like injecting hooks and etc.
Since this is based on some heuristic - you're not guaranteed (generally speaking) to cover all possible scenarios. In some circumstances you may miss the changes.
(4)
Hook at lower level: intercept calls to the drawing functions. Since there's enormous number of such functions in the user mode - the only realistic possibility of doing this is in the kernel mode.
You may write a virtual video driver (either "mirror" video driver, or hook the existing one) to receive all the drawing in the system. Then whenever you receive a drawing request on the specific area - you'll know it's changed.
Pros:
Lower CPU usage.
100% guarantee to intercept all drawings, without heuristics
Somewhat cleaner - no need to inject hooks into apps/controls
Cons:
It's a driver development! Unless you're experienced in it - it's a real nightmare.
More complex installation. Need administrator rights, most probably need restart.
Still considerable CPU load and bandwidth
(5)
Going on with driver development. As long as you know now which drawing functions are called - you may switch the strategy now. Instead of "remembering" dirty areas and polling the screen there - you may just "remember" the drawing function invoked with all the parameters, and then "repeat" it at the host side.
By such you don't have to poll the screen at all. You work in a "vectored" method (as opposed to "raster").
This however is much more complex to implement. Some drawing functions take as parameters another bitmaps, which in turn are drawn using another drawing functions and etc. You'll have to spy for bitmaps as well as screen.
Pros:
Zero CPU load
Best possible network traffic
Guaranteed to work always
Cons:
It's a driver development at its best! Months of development are guaranteed
Requires state-of-the-art programming, deep understanding of 2D drawing
Need to write the code at host which will "draw" all the "Recorded" commands.