Mission Control and Exposé are capable of displaying a live (usually scaled down) preview of an NSWindow. I want to do the same thing, or at least figure out exactly how they do. Decompiling Mission Control binary revealed very little, which leads me to believe that the significant logic is somewhere else (given the multiple references to "com.apple.windowserver," likely closely coupled with the window server).
Related
When a window is resized by aero snap, User32.GetWindowPlacement(hWnd).rcNormalPosition still stores its original rectangle, while User32.GetWindowRect is affected.
Since aero snap seems independent from WINDOWPLACEMENT, now we cannot collect the complete information of the actual placement simply using user32.dll. Thus I'm wondering if there's a way to get the aero snap state of a window, indicating whether the window is docked and which side the window is docked to.
Aero Snap is a feature of the Shell, not the windowing system. Thus, the windowing system cannot provide that information, because it is not aware of those states.
And the Shell doesn't make this information available either. So, in essence, the system doesn't provide the Aero Snap state of any given window through a public API.
I like having my main windows remember all of their placement information so that I it can be restored when they are restarted. In that past, it was enough to save a copy of the window placement structure and to set it back when recreating the window.
The introduction of snap required keeping some extra information. I detected whether a window appeared to be snapped by comparing its window rectangle to the work area rectangle of the monitor that contains the window. If it seemed to be snapped to one of the edges, I recorded that along with the placement information. Upon creating the window, I first restore the window placement, and then, if I have a snap state recorded, I change the window's size and position accordingly.
You can distinguish between a window that's been snapped to a monitor edge from one that's been carefully sized and placed there because the snapped window's rectangle won't match the one in the window placement.
This approach worked great in Windows 7. I recently discovered that Windows 10 added more flexibility to the snap locations and sizes as well as playing more games to achieve the annoyingly invisible resize borders. So my detection code doesn't always recognize a snapped window, but that should fixable.
Some popular software (like the major browsers) seem to remember that they were snapped, and I assume they use the same approach.
One of the things that I've noticed (at least on Windows anyway), is that the mouse cursor is drawn with much less latency than even standard Windows elements.
A good example of this would be to start dragging on the desktop. You can easily notice that the drag rectangle is lagging significantly behind the cursor.
My first question is: why is this the case?
I can't imagine drawing a rectangle being so much more expensive than drawing the cursor. Certainly not by a frame or two.
And my second question is, would it ever be possible to match one's application rendering 1:1 with cursor input?
A good use case for this would be either this selection rectangle, or drag previews for draggable items. Both of which lag behind quite significantly from the OS mouse pointer (independent of any framework or library used).
Selecting icons on the desktop with the selection rectangle is not that slow on my system (DWM on), it is lagging a little bit but not enough for me to really care.
The "Show Window Contents while Dragging" option has always been rather slow which is why it was not on by default in older Windows versions.
The mouse cursor on the other hand can be rendered directly by your hardware. That is, Windows sends the cursor image to your graphics card and after that Windows only has to tell the graphics card the cursor position and this is much faster than all the messages and user/kernel context switches involved when you resize and paint a window. The mouse driver probably uses hardware interrupts/timers with a higher priority than your normal software as well.
You can try to disable hardware cursors with a registry hack but the HID/mouse driver and the raw input thread in win32k will still have a higher priority than your application.
I'm trying to implement a cross-platform UI library that takes as little system resource as possible. I'm considering to either use my own software renderer or opengl.
For stationary controls everything's fine, I can repaint only when it's needed. However, when it comes to implementing animations, especially animated blinking carets like the 'phase' caret in sublime text, I don't see a easy way to balance resource usage and performance.
For a blinking caret, it's required that the caret be redrawn very frequently(15-20 times per sec at least, I guess). On one hand, the software renderer supports partial redraw but is far too slow to be practical(3-4 fps for large redraw regions, say, 1000x800, which makes it impossible to implement animations). On the other hand, opengl doesn't support partial redraw very well as far as I know, which means the whole screen needs to be rendered at 15-20 fps constantly.
So my question is:
How are carets usually implemented in various UI systems?
Is there any way to have opengl to render to only a proportion of the screen?
I know that glViewport enables rendering to part of the screen, but due to double buffering or other stuff the rest of the screen is not kept as it was. In this way I still need to render the whole screen again.
First you need to ask yourself.
Do I really need to partially redraw the screen?
OpenGL or better said the GPU can draw thousands of triangles at ease. So before you start fiddling with partial redrawing of the screen, then you should instead benchmark and see whether it's worth looking into at all.
This doesn't however imply that you have to redraw the screen endlessly. You can still just redraw it when changes happen.
Thus if you have a cursor blinking every 500 ms, then you redraw once every 500 ms. If you have an animation running, then you continuously redraw while that animation is playing (or every time the animation does a change that requires redrawing).
This is what Chrome, Firefox, etc does. You can see this if you open the Developer Tools (F12) and go to the Timeline tab.
Take a look at the following screenshot. The first row of the timeline shows how often Chrome redraws the windows.
The first section shows a lot continuously redrawing. Which was because I was scrolling around on the page.
The last section shows a single redraw every few 500 ms. Which was the cursor blinking in a textbox.
Open the image in a new tab, to see better what's going on.
Note that it doesn't tell whether Chrome is fully redrawing the window or only that parts of it. It is just showing the frequency of the redrawing. (If you want to see the redrawn regions, then both Firefox and Chrome has "Show Paint Rectangles".)
To circumvent the problem with double buffering and partially redrawing. Then you could instead draw to a framebuffer object. Now you can utilize glScissor() as much as you want. If you have various things that are static and only a few dynamic things. Then you could have multiple framebuffer objects and only draw the static contents once and continuously update the framebuffer containing the dynamic content.
However (and I can't emphasize this enough) benchmark and check if this is even needed. Having two framebuffer objects could be more expensive than just always redrawing everything. The same goes for say having a buffer for each rectangle, in contrast to packing all rectangles in a single buffer.
Lastly to give an example let's take NanoGUI (a minimalistic GUI library for OpenGL). NanoGUI continuously redraws the screen.
The problem with not just continuously redrawing the screen is that now you need a system for issuing a redraw. Now calling setText() on a label needs to callback and tell the window to redraw. Now what if the parent panel the label is added to isn't visible? Then setText() just issued a redundant redrawing of the screen.
The point I'm trying to make is that if you have a system for issuing redrawing of the screen. Then that might be more prone to errors. Thus unless continuously redrawing is an issue, then that is definitely a more optimal starting point.
I was reading Larry Osterman's latest blog post about debugging a flickering problem in the Windows Vista/7 volume control, and I suddenly realized that I can't recall ever seeing an application flicker on my OS X laptop. Even applications that otherwise seem to be poorly written avoid the flicker problem in my experience. Without this turning into an Apple vs Windows debate (please), why do OS X applications not seem to have the same flickering problem?
I have trouble believing that Apple developers are simply amazing at programming flicker-free GUIs, while Windows programmers suck, so what's the reason? Does the OS X API require all GUIs to implement double-buffering? While some apps have the slightly sluggish double-buffered resize behavior, many don't, and they still avoid flickering. Is the OS X repaint flow somehow fundamentally different from Windows, avoiding the WM_ERASEBKGRND problem entirely? Or is there some other possibility that I'm not seeing?
Update: Thank you for your answers. I wish I could select both ken and cb160's answers, because they are both helpful.
Mac OS X has double buffered windows.
You don't have to do anything to make it happen. It's behind the scenes.
You (almost always) don't explicitly draw to a window in Cocoa when something changes, you invalidate a region of the window. The framework will later descend the hierarchy of views and draw the dirty regions of the window into a secondary buffer. Then it swaps the buffers.
You can optionally make some promises that allow the framework to take shortcuts when redrawing, but they're all opt-in. Only savvy views are affected.
If your subclass of NSView implements the isOpaque method to return YES, then the framework will never clear anything behind your view or draw any of the views under it.
Implementing preservesContentDuringLiveResize to return YES gives you some extra responsibilities, but can improve performance during window resizing.
10.6 added another two new APIs of this sort, layerContentsRedrawPolicy and layerContentsPlacement.
Last, custom drawing is less common than on Windows. The majority of views you see are framework-supplied and not subclassed. Framework-supplied means optimized-by-apple.
Both Windows Vista/7 and OSX use compositing engines to draw rasterised bitmaps on the screen. These compositing engines are responsible for processing output from all windows and drawing the final screen image. This compositing approach is how OSX is able to use the genie effect when minimizing to the dock and how aero draws the translucent borders. They also prevent flickering as if the bitmap to fill a particular area of the screen is not available, it will use the image it has already rather than drawing a blank region.
OSX has had a compositing engine since it first shipped. At the time, lots of people though this was a crazy appraoch as all the video cards shipping at the time wer optimized to draw bitmaps (ie, windows buttons and borders) and not composited images. In later versions of OSX, the compositing was pushed off to the GPU (in Quartz Extreme)and so took significant load off of the CPU and made more effects possible.
Because the Windows compositer was only added in windows Vista and then only when there was a GPU available and you had the right version of the OS, it is not as pervassive as the Quartz Compositer in OSX. Because the compositer is not always used in Windows, flickering will occur when a region is blanked and the application responsible for drawing is not able to redraw the region qucikly enough.
Yup, it's all double buffered automagically. Of course, if you are running
legacy code from mac os 9, or code ported from windoze, that mean's you're
probably triple buffering without knowing it. Hey, cycles are cheap!
The performance of a Direct3D application seems to be significantly better in full screen mode compared to windowed mode. What are the technical reasons behind this?
I guess it has something to do with the fact that a full screen application can gain exclusive control for the display. But why the application cannot gain exclusive control for part of the screen (i.e. window) and have the same performance benefits?
Here are the cliff notes on how things work underneath.
Monitor screen always needs to be associated with so-called primary surface to be able to display anything, i.e. videocard can only scan out of one surface in video memory.
When application is fullscreen (and everything was set up correctly to enable flipping), primary surface is just one of the application backbuffers, and flipped to another backbuffer every frame. It is the most efficient way of presenting on the screen, but it requires application to own the entire monitor area (i.e. entire primary surface).
When there's no fullscreen application and DWM is off, primary surface is owned by OS, and every windowed application performs a blit from application backbuffer to a primary surface. This blit takes some GPU time to complete (as well as blits from the other applications visible on the screen), so it's not as efficient as fullscreen presentation. XP worked that way.
When DWM is composing the screen, things get even more complicated.
Here, DWM owns the primary surface and needs to draw application windows there. To make it possible, every window has an associated surface holding its contents, called redirection surface (which allows DWM to enable window ghosting, glass effects, and all that good stuff). Every time D3D application issues a frame, it adds a blit to a redirection surface.
That way, several blits need to happen: blit to a redirection surface by the app, blit from a redirection surface to the primary by DWM, which is, again, some overhead compared to fullscreen.
Note all of that additional work is on the GPU, so it doesn't affect CPU performance.
Stuff to read further:
http://blogs.msdn.com/greg_schechter/archive/2006/03/19/555087.aspx
http://blogs.msdn.com/greg_schechter/archive/2006/05/02/588934.aspx
http://blogs.msdn.com/greg_schechter/archive/2006/03/05/544314.aspx
There's a bit on MSDN that says full screen mode uses buffer flipping, if set up correctly, as opposed to blitting. It makes sense.
Of course you can (and in a way, do) give exclusive control for part of the screen to an application, but what happens to the rest of the screen? You still have to blit, do occlusion checking, etc. on the rest of the windows, and I think that's what causes the performance hit.
I'll add to #aib's answer that the rest of the screen is being managed by the OS. So, if anything else needs to be drawn/worked upon simultaneously, there has to be a performance hit.
For example, if you have a video playing in Windows Media Player in one window, then start Civilization in another, when Civ starts doing its fancy graphics, it will need to share screen space with everything else (like the video.
Whereas if the DirectX app has the full-screen, everything else might be "updating" or "playing", but not being drawn.
Basically, the video hardware is completely dedicated to the exclusive mode application.
There is no contention for video resources (pipeline, texture memory, etc...)
In particular, texture upload can be a big bottleneck. The less you have to do it (because you have it all), the better.