THE PROBLEM: When I grab the resize border of my Windows app, especially the top or left borders, and resize the window, the contents of the window do resize "live" as I drag, but they resize in a hideous manner that looks like a blatant bug to even the most novice user: the contents at the opposite edge of the window from the edge I am dragging jitter/flicker/jump back and forth wildly. Depending on the situation, the phenomenon may look like:
contents that seem to walk off the edge of the window and snap back when we slow down or stop dragging
contents that seem to pull into the window, intermittently displaced by a border of varying colors, often black or white
a seriously ugly "double image" with two overlapping copies of the content displaced by a distance proportional to how much/how fast we are dragging
The ugly phenomenon stops as soon as I stop dragging, but during the dragging it makes the app look amateurish and unprofessional.
It is not an understatement to say this Windows problem has driven thousands of app developers crazy.
Here are two example pictures of the phenomenon, kindly prepared for a related question by Roman Starkov:
Jitter:
Border:
Another example showing the evil "double image" phenomenon (note the quick flash) from Kenny Liu:
Another example video of the phenomenon with Task Manager is here.
THE QUESTION: Any developer who has experienced this problem quickly finds that there are at least 30 Stack Overflow questions, some recent and some dating from 2008, full of promising-sounding answers that rarely work. The reality is that this one problem has many causes, and the existing Stack Overflow questions/answers never make the wider context clear. This question seeks to answer:
what are the most likely causes of this kind of ugly jitter/flicker/jumping?
how do I tell which cause I am seeing?
is this cause specific to particular graphics drivers or general for Windows?
how do I fix each cause? can an app fix it?
(This is meant as a canonical Q&A to explain all the different causes of window resize jitter so that users can identify which of the causes is causing their problem and solve it. As the answers explain, all the permutations above (native/managed, window/dialog, XP-10) boil down to only two root causes, but identifying which you have is the tricky part.)
SCOPE OF THIS QUESTION: For the scope of this question, the phenomenon happens with:
both native Win32 and managed .NET/WPF/Windows Forms apps
both normal Win32 windows and Win32 Dialog windows
Windows versions including XP, Vista, 7, 8, and 10 (but see below for the dark truth of multiple causes)
NOT IN SCOPE OF THIS QUESTION:
If your app has one or more child windows (child HWNDs), the info in this question is useful to you (since the jerk-causing BitBlts we will describe are applied to your child windows along with the parent window), but during window resize you have an additional problem to handle that is beyond the scope of this question: you need to make all your child windows move atomically and in sync with the parent window. For this task, you will probably want BeginDeferWindowPos/DeferWindowPos/EndDeferWindowPos and you can find out about them here and here.
This question assumes that if your app draws to a window using GDI, DirectX, or OpenGL, then you have already implemented a WM_ERASEBKGND handler in your wndproc that simply returns 1. WM_ERASEBKGND is an arcane Windows remnant from Windows 3.1 that comes before WM_PAINT to give your app a chance to "erase the background" of your window before you draw your window...uh huh. If you let the WM_ERASEBKGND message go into DefWindowProc(), that will cause your entire window to get painted a solid color, usually white, on each redraw, including redraws that happen during live window resizing. The result is an ugly full-window flicker that is gross, but not the type of jitter/flicker/jumping we are talking about in this question. Intercepting WM_ERASEBKGND fixes this problem immediately.
This question is primarily about live-resize by dragging window borders with the mouse. However, much of what is written here also applies to ugly artifacts you can see when an app manually does a one-time window resize using SetWindowPos(). These are less visible though because they only flick on the screen for one instant, rather than over a long period of dragging.
This question is not about how to make your app-specific drawing code go faster, even though doing so may be a solution to the ugly resizing problem in many cases. If your app really does take huge amounts of time to redisplay its contents during live window resize, consider optimizing your drawing code in general or at least switching to a faster, lower-quality drawing mode during resize by intercepting the WM_ENTERSIZEMOVE/WM_EXITSIZEMOVE messages to detect resize.
If your app fails to resize at all during app resizing (e.g. it "hangs" during resizing, especially if it is OpenGL using GLFW or other library), see these other questions which explain about Microsoft's hideous nested/modal event loop inside WM_SYSCOMMAND during dragging: here especially this good answer, here, here, here, and here.
PART 2: Identifying and Fixing Windows Resize Problems
Note: you want to read PART 1 first for this answer to make sense.
This answer will not solve all your resizing problems.
It organizes the still-usable ideas from other posts and adds a few novel ideas.
None of this behavior is at all documented on Microsoft's MSDN, and what follows below is the result of my own experimentation and looking at other StackOverflow posts.
2a. Resize Problems from SetWindowPos() BitBlt and Background Fill
The following problems happen on all versions of Windows. They date back to the very first days of live-scrolling on the Windows platform (Windows XP) and are still present on Windows 10. On more recent Windows versions, other resize problems may layer on top of this problem, as we explain below.
Here are the Windows events associated with a typical session of clicking a window border and dragging that border. Indentation indicates nested wndproc (nested because of sent (not posted) messages or because of the hideous Windows modal event loop mentioned in "NOT IN SCOPE OF THIS QUESTION" in the question above):
msg=0xa1 (WM_NCLBUTTONDOWN) [click mouse button on border]
msg=0x112 (WM_SYSCOMMAND) [window resize command: modal event loop]
msg=0x24 (WM_GETMINMAXINFO)
msg=0x24 (WM_GETMINMAXINFO) done
msg=0x231 (WM_ENTERSIZEMOVE) [starting to size/move window]
msg=0x231 (WM_ENTERSIZEMOVE) done
msg=0x2a2 (WM_NCMOUSELEAVE)
msg=0x2a2 (WM_NCMOUSELEAVE) done
loop:
msg=0x214 (WM_SIZING) [mouse dragged]
msg=0x214 (WM_SIZING) done
msg=0x46 (WM_WINDOWPOSCHANGING)
msg=0x24 (WM_GETMINMAXINFO)
msg=0x24 (WM_GETMINMAXINFO) done
msg=0x46 (WM_WINDOWPOSCHANGING) done
msg=0x83 (WM_NCCALCSIZE)
msg=0x83 (WM_NCCALCSIZE) done
msg=0x85 (WM_NCPAINT)
msg=0x85 (WM_NCPAINT) done
msg=0x14 (WM_ERASEBKGND)
msg=0x14 (WM_ERASEBKGND) done
msg=0x47 (WM_WINDOWPOSCHANGED)
msg=0x3 (WM_MOVE)
msg=0x3 (WM_MOVE) done
msg=0x5 (WM_SIZE)
msg=0x5 (WM_SIZE) done
msg=0x47 (WM_WINDOWPOSCHANGED) done
msg=0xf (WM_PAINT) [may or may not come: see below]
msg=0xf (WM_PAINT) done
goto loop;
msg=0x215 (WM_CAPTURECHANGED) [mouse released]
msg=0x215 (WM_CAPTURECHANGED) done
msg=0x46 (WM_WINDOWPOSCHANGING)
msg=0x24 (WM_GETMINMAXINFO)
msg=0x24 (WM_GETMINMAXINFO) done
msg=0x46 (WM_WINDOWPOSCHANGING) done
msg=0x232 (WM_EXITSIZEMOVE)
msg=0x232 (WM_EXITSIZEMOVE) done [finished size/moving window]
msg=0x112 (WM_SYSCOMMAND) done
msg=0xa1 (WM_NCLBUTTONDOWN) done
Each time you drag the mouse, Windows gives you the series of messages shown in the loop above. Most interestingly, you get WM_SIZING then WM_NCCALCSIZE then WM_MOVE/WM_SIZE, then you may (more on that below) receive WM_PAINT.
Remember we assume you have provided a WM_ERASEBKGND handler that returns 1 (see "NOT IN SCOPE OF THIS QUESTION" in the question above) so that message does nothing and we can ignore it.
During the processing of those messages (shortly after WM_WINDOWPOSCHANGING returns), Windows makes an internal call to SetWindowPos() to actually resize the window. That SetWindowPos() call first resizes the non-client area (e.g. the title bars and window border) then turns its attention to the client area (the main part of the window that you are responsible for).
During each sequence of messages from one drag, Microsoft gives you a certain amount of time to update the client area by yourself.
The clock for this deadline apparently starts ticking after WM_NCCALCSIZE returns. In the case of OpenGL windows, the deadline is apparently satisfied when you call SwapBuffers() to present a new buffer (not when your WM_PAINT is entered or returns). I do not use GDI or DirectX, so I don't know what the equavalent call to SwapBuffers() is, but you can probably make a good guess and you can verify by inserting Sleep(1000) at various points in your code to see when the behaviors below get triggered.
How much time do you have to meet your deadline? The number seems to be around 40-60 milliseconds by my experiments, but given the kinds of shenanigans Microsoft routinely pulls, I wouldn't be surprised if the number depends on your hardware config or even your app's previous behavior.
If you do update your client area by the deadline, then Microsoft will leave your client area beautifully unmolested. Your user will only see the pixels that you draw, and you will have the smoothest possible resizing.
If you do not update your client area by the deadline, then Microsoft will step in and "help" you by first showing some other pixels to your user, based on a combination of the "Fill in Some Background Color" technique (Section 1c3 of PART 1) and the "Cut off some Pixels" technique (Section 1c4 of PART 1). Exactly what pixels Microsoft shows your user is, well, complicated:
If your window has a WNDCLASS.style that includes the CS_HREDRAW|CS_VREDRAW bits (you pass the WNDCLASS structure to RegisterClassEx):
Something surprisingly reasonable happens. You get the logical behavior shown in Figures 1c3-1, 1c3-2, 1c4-1, and 1c4-2 of PART 1. When enlarging the client area, Windows will fill in newly exposed pixels with the "background color" (see below) on the same side of the window you are dragging. If needed (left and top border cases), Microsoft does a BitBlt to accomplish this. When shrinking the client area, Microsoft will chop off pixels on the same side of the window you are dragging. This means you avoid the truly heinous artifact that makes objects in your client area appear to move in one direction then move back in the other direction.
This may be good enough to give you passable resize behavior, unless you really want to push it and see if you can totally prevent Windows from molesting your client area before you have a chance to draw (see below).
Do not implement your own WM_NCCALCSIZE handler in this case, to avoid buggy Windows behavior described below.
If your window has a WNDCLASS.style that does not include the CS_HREDRAW|CS_VREDRAW bits (including Dialogs, where Windows does not let you set WNDCLASS.style):
Windows tries to "help" you by doing a BitBlt that makes a copy of a certain rectangle of pixels from your old client area and writes that rectangle to a certain place in your new client area. This BitBlt is 1:1 (it does not scale or zoom your pixels).
Then, Windows fills in the other parts of the new client area (the parts that Windows did not overwrite during the BitBlt operation) with the "background color."
The BitBlt operation is often the key reason why resize looks so bad. This is because Windows makes a bad guess about how your app is going to redraw the client area after the resize. Windows places your content in the wrong location. The net result is that when the user first sees the BitBlt pixels and then sees the real pixels drawn by your code, your content appears to first move in one direction, then jerk back in the other direction. As we explained in PART 1, this creates the most hideous type of resize artifact.
So, most solutions for fixing resize problems involve disabling the BitBlt.
If you implement a WM_NCCALCSIZE handler and that handler returns WVR_VALIDRECTS when wParam is 1, you can actually control which pixels Windows copies (BitBlts) from the old client area and where Windows places those pixels in the new client area. WM_NCCALCSIZE is just barely documented, but see the hints about WVR_VALIDRECTS and NCCALCSIZE_PARAMS.rgrc[1] and [2] in the MSDN pages for WM_NCCALCSIZE and NCCALCSIZE_PARAMS. You can even provide NCCALCSIZE_PARAMS.rgrc[1] and [2] return values that completely prevent Windows from BitBlting any of the pixels of the old client area to the new client area, or cause Windows to BitBlt one pixel from and to the same location, which is effectively the same thing since no on-screen pixels would get modified. Just set both NCCALCSIZE_PARAMS.rgrc[1] and [2] to the same 1-pixel rectangle. In combination with eliminating the "background color" (see below), this gives you a way to prevent Windows from molesting your window's pixels before you have time to draw them.
If you implement a WM_NCCALCSIZE handler and it returns anything other than WVR_VALIDRECTS when wParam is 1, then you get a behavior which (at least on Windows 10) does not at all resemble what MSDN says. Windows seems to ignore whatever left/right/top/bottom alignment flags you return. I advise you do not do this. In particular the popular StackOverflow article How do I force windows NOT to redraw anything in my dialog when the user is resizing my dialog? returns WVR_ALIGNLEFT|WVR_ALIGNTOP and this appears to be completely broken now at least on my Windows 10 test system. The code in that article might work if it is changed to return WVR_VALIDRECTS instead.
If you do not have your own custom WM_NCCALCSIZE handler, you get a pretty useless behavior that is probably best avoided:
If you shrink the client area, nothing happens (your app gets no WM_PAINT at all)! If you're using the top or left border, your client area contents will move along with the top left of the client area. In order to get any live resizing when shrinking the window, you have to manually draw from a wndproc message like WM_SIZE, or call InvalidateWindow() to trigger a later WM_PAINT.
If you enlarge the client area
If you drag the bottom or right window border, Microsoft fills in the new pixels with the "background color" (see below)
If you drag the top or left window border, Microsoft copies the existing pixels to the top left corner of the expanded window and leaves an old junk copy of old pixels in the newly opened space
So as you can see from this sordid tale, there appear to be two useful combinations:
2a1. WNDCLASS.style with CS_HREDRAW|CS_VREDRAW gives you the behavior in Figures 1c3-1, 1c3-2, 1c4-1, and 1c4-2 of PART 1, which is not perfect but at least your client area content will not move one direction then jerk back in the other direction
2a2. WNDCLASS.style without CS_HREDRAW|CS_VREDRAW plus a WM_NCCALCSIZE handler returning WVR_VALIDRECTS (when wParam is 1) that BitBlts nothing, plus disabling the "background color" (see below) may completely disable Windows' molestation of your client area.
There is apparently another way to achieve the effect of combination 2a2. Instead of implementing your own WM_NCCALCSIZE, you can intercept WM_WINDOWPOSCHANGING (first passing it onto DefWindowProc) and set WINDOWPOS.flags |= SWP_NOCOPYBITS, which disables the BitBlt inside the internal call to SetWindowPos() that Windows makes during window resizing. I have not tried this trick myself but many SO users reported it worked.
At several points above, we mentioned the "background color." This color is determined by the WNDCLASS.hbrBackground field that you passed to RegisterClassEx. This field contains an HBRUSH object. Most people set it using the following boilerplate code:
wndclass.hbrBackground = (HBRUSH)(COLOR_WINDOW+1);
The COLOR_WINDOW+1 incantation gives you a white background color. See MSDN dox for WNDCLASS for the +1 explanation and note there is a lot of wrong info about the +1 on StackOverflow and MS forums.
You can choose your own color like this:
wndclass.hbrBackground = CreateSolidBrush(RGB(255,200,122));
You can also disable the background fill-in using:
wndclass.hbrBackground = NULL;
which is another key ingredient of combination 2a2 above. But be aware that newly uncovered pixels will take on some essentially random color or pattern (whatever garbage happens to be in your graphics framebuffer) until your app catches up and draws new client area pixels, so it might actually be better to use combination 2a1 and choose a background color that goes with your app.
2b. Resize Problems from DWM Composition Fill
At a certain point during the development of Aero, Microsoft added another live resize jitter problem on top of the all-Windows-version problem described above.
Reading earlier StackOverflow posts, it is actually hard to tell when this problem was introduced, but we can say that:
this problem definitely occurs in Windows 10
this problem almost certainly occurs in Windows 8
this problem may have also occurred in Windows Vista with Aero enabled (many posts with resize problems under Vista do not say if they have Aero enabled or not).
this problem probably did not occur under Windows 7, even with Aero enabled.
The problem revolves around a major change of architecture that Microsoft introduced in Windows Vista called DWM Desktop Composition. Applications no longer draw directly to the graphics framebuffer. Instead, all applications are actually drawing into an off-screen framebuffer which is then composited with the output of other apps by the new, evil Desktop Window Manager (DWM) process of Windows.
So, because there is another process involved in displaying your pixels, there is another opportunity to mess up your pixels.
And Microsoft would never miss such an opportunity.
Here is what apparently happens with DWM Compostion:
The user clicks the mouse on a window border and begins to drag the mouse
Each time the user drags the mouse, this triggers the sequence of wndproc events in your application that we described in section 2a above.
But, at the same time, DWM (which remember is a separate process that is runnning asynchronously to your app) starts its own deadline timer.
Similarly to section 2a above, the timer apparently starts ticking after WM_NCCALCSIZE returns and is satisfied when your app draws and calls SwapBuffers().
If you do update your client area by the deadline, then DWM will leave your client area beautifully unmolested. There is still a definite chance that your client area could still get molested by the problem in section 2a, so be sure to read section 2a as well.
If you do not update your client area by the deadline, then Microsoft will do something truly hideous and unbelievably bad (didn't Microsoft learn their lesson?):
Suppose this is your client area before the resize, where A, B, C, and D represent pixel colors at the middle of your client area top, left, right, and bottom edges:
--------------AAA-----------------
| |
B C
B C
B C
| |
--------------DDD-----------------
Suppose you are using the mouse to enlarge your client area in both dimensions. Genius Windows DWM (or perhaps Nvidia: more on that later) will always copy the pixels of your client area to the upper-left corner of the new client area (regardless of which window border you are dragging) and then do the most absurd thing imaginable to the rest of the client area. Windows will take whatever pixel values used to be along the bottom edge of your client area, stretch them out to the new client area width (a terrible idea we explored in Section 1c2 of PART 1, and replicate those pixels to fill in all the newly opened space at the bottom (see what happens to D). Then Windows will take whatever pixel values used to be along the right edge of your client area, stretch them out to the new client area height, and replicate them to fill in the newly opened space at the top-right:
--------------AAA-----------------------------------------------
| | |
B C |
B C |
B CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
| |CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
--------------DDD-----------------CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
| DDDDDDDDD |
| DDDDDDDDD |
| DDDDDDDDD |
| DDDDDDDDD |
| DDDDDDDDD |
------------------------------DDDDDDDDD-------------------------
I cannot even imagine what they were smoking. This behavior produces the worst possible result in many cases. First, it's almost guaranteed to generate the horrific back-and-forth motion we showed in Figure 1c3-3 and Figure 1c4-3 of PART 1 when dragging the left and top window borders, since the rectangle copied is always at the upper-left regardless of which window border you are dragging. Second, the even more ridulous thing that's happening with the edge pixels being replicated is going to produce ugly bars if you happen to have any pixels set there other than the background color. Notice how the bars of C and D created do not even line up with the original C and D from the copied old pixels. I can understand why they are replicating the edge, hoping to find background pixels there to "automate" the process of background color detection, but it seems the likelihood of this actually working is heavily outweighed by the hack factor and chance of failure. It would be better if DWM used the app's chosen "background color" (in WNDCLASS.hbrBackground), but I suspect DWM might not have access to that info since DWM is in a different process, hence the hack. Sigh.
But we haven't even gotten to the worst part yet:
What actually is the deadline that DWM gives you to draw your own client area before DWM corrupts it with this clumsy hack of a guess? Apparently (from my experiments) the deadline is on the order of 10-15 milliseconds! Given that 15 milliseconds is close to 1/60, I would guess that the deadline is actually the end of the current frame. And the vast majority of apps are unable to meet this deadline most of the time.
That is why, if you launch Windows Explorer on Windows 10 and drag the left border, you will most likely see the scroll bar on the right jitter/flicker/jump around erratically as if Windows were written by a fourth grader.
I cannot believe that Microsoft has released code like this and considers it "done." It is also possible that the responsible code is in the graphics driver (e.g. Nvidia, Intel, ...) but some StackOverflow posts led me to believe that this behavior is cross-device.
There is very little you can do to prevent this layer of incompetence from generating hideous jitter/flicker/jump when resizing using the left or top window border. That is because the rude, non-consentual modification of your client area is happening in another process.
I am really hoping that some StackOverflow user will come up with some magic DWM setting or flag in Windows 10 that we can make to either extend the deadline or disable the horrible behavior completely.
But in the meantime, I did come up with one hack that somewhat reduces the frequency of the hideous back-and-forth artifacts during window resize.
The hack, inspired by a comment in https://stackoverflow.com/a/25364123/1046167 , is to do a best-effort at synchronizing the app process with the vertical retrace that drives DWM's activity. Actually making this work in Windows is not trivial. The code for this hack should be the very last thing in your WM_NCCALCSIZE handler:
LARGE_INTEGER freq, now0, now1, now2;
QueryPerformanceFrequency(&freq); // hz
// this absurd code makes Sleep() more accurate
// - without it, Sleep() is not even +-10ms accurate
// - with it, Sleep is around +-1.5 ms accurate
TIMECAPS tc;
MMRESULT mmerr;
MMC(timeGetDevCaps(&tc, sizeof(tc)), {});
int ms_granularity = tc.wPeriodMin;
timeBeginPeriod(ms_granularity); // begin accurate Sleep() !
QueryPerformanceCounter(&now0);
// ask DWM where the vertical blank falls
DWM_TIMING_INFO dti;
memset(&dti, 0, sizeof(dti));
dti.cbSize = sizeof(dti);
HRESULT hrerr;
HRC(DwmGetCompositionTimingInfo(NULL, &dti), {});
QueryPerformanceCounter(&now1);
// - DWM told us about SOME vertical blank
// - past or future, possibly many frames away
// - convert that into the NEXT vertical blank
__int64 period = (__int64)dti.qpcRefreshPeriod;
__int64 dt = (__int64)dti.qpcVBlank - (__int64)now1.QuadPart;
__int64 w, m;
if (dt >= 0)
{
w = dt / period;
}
else // dt < 0
{
// reach back to previous period
// - so m represents consistent position within phase
w = -1 + dt / period;
}
// uncomment this to see worst-case behavior
// dt += (sint_64_t)(0.5 * period);
m = dt - (period * w);
assert(m >= 0);
assert(m < period);
double m_ms = 1000.0 * m / (double)freq.QuadPart;
Sleep((int)round(m_ms));
timeEndPeriod(ms_granularity);
You can convince yourself that this hack is working by uncommenting the line that shows "worst-case" behavior by trying to schedule the drawing right in the middle of a frame rather than at vertical sync, and noticing how many more artifacts you have. You can also try varying the offset in that line slowly and you will see that artifacts abruptly disappear (but not completely) at about 90% of the period and come back again at about 5-10% of the period.
Since Windows is not a real-time OS, it is possible for your app to be
preempted anywhere in this code, leading to inaccuracy in the pairing of now1 and dti.qpcVBlank. Preemption in this small code section is rare, but possible. If you want, you can compare now0 and now1 and loop around again if the bound is not tight enough. It is also possible for preemption to disrupt the timing of Sleep() or the code before or after Sleep(). There's not much you can do about this, but it turns out timing errors in this part of the code are swamped by the uncertian behavior of DWM; you are still going to get some window resize artifacts even if your timing is perfect. It's just a heuristic.
There is a second hack, and it is an incredibly creative one: as explained in the StackOverflow post Can't get rid of jitter while dragging the left border of a window, you can actually create two main windows in your app, and every time Windows would do SetWindowPos, you intecept that and instead hide one window and show the other! I haven't tried this yet but the OP reports that it bypasses the insane pixel DWM pixel copy described above.
There is a third hack, which might work depending on your application (especially in combination with the timing hack above). During live resizing (which you can detect by intercepting WM_ENTERSIZEMOVE/WM_EXITSIZEMOVE), you could modify your drawing code to initially draw something much simpler that is much more likely to complete within the deadline imposed by problem 2a and 2b, and call SwapBuffers() to claim your prize: that will be enough to prevent Windows from doing the bad blit/fill described in section 2a and 2b. Then, immediately after the partial draw, do another draw that fully updates the window contents and call SwapBuffers() again. That might still look somewhat odd, since the user will see your window update in two parts, but it's likely to look much better than the hideous back-and-forth motion artifact from Windows.
One more tantalizing point: some apps in Windows 10, including the console (start cmd.exe), are rock-solid free of DWM Composition artifacts even when dragging the left border. So there is some way of bypassing the problem. Let's find it!
2c. How to Diagnose Your Problem
As you try to solve your particular resize problem, you may wonder which of the overlapping effects from Section 2a and Section 2b you are seeing.
One way to separate them is to debug on Windows 7 (with Aero disabled, just to be safe) for a bit.
Another way to quickly identify if you are seeing the problem in Section 2b is to modify your app to display the test pattern described in Section 2b, like this example (note the 1-pixel-thin colored lines on each of the four edges):
Then grab any window border and start resizing that border rapidly. If you see intermittent giant colored bars (blue or green bars in the case of this test pattern, since there is blue on the bottom edge and green on the right edge) then you know you are seeing the problem in Section 2b.
You can test if you are seeing the problem in Section 2a by setting WNDCLASS.hbrBackground to a distinct background color, like red. As you resize the window, newly exposed parts will show up with that color. But read through Section 2a to make sure your message handlers are not causing Windows to BitBlt the entire client area, which would cause Windows not to draw any background color.
Remember that the problems in Section 2a and 2b only show up if your app fails to draw by a certain deadline, and each problem has a different deadline.
So, without modification, your app might show the Section 2b problem only, but if you modify your app to draw more slowly (insert Sleep() in WM_PAINT before SwapBuffers() for example), you may miss the deadline for both Section 2a and Section 2b and start to see both problems simultaneously.
This may also happen when you change your app between a slower DEBUG build and a RELEASE build, which can make chasing these resize problems very frustrating. Knowing what's going on under the hood can help you deal with the confusing results.
PART 1: What Makes Resize Look Good or Bad?
There is so much ambiguity and unclarity in StackOverflow questions about smooth resize that we need to establish a common vocabulary to help people make their answers more clear.
That is what we will do in this section.
To keep things simple, we going to explain the problems of smooth resizing in the horizontal dimension only, but everything here applies to vertical resizing just the same.
Below we will refer to a window's
"non-client area:" the part of the window that Windows manages, including the title bar at the top and window borders around all edges, and
"client area:" the main part of the window that you are responsible for
Suppose you have an app with:
a button or label L that is supposed to remain flush-left
a button or label R that is supposed to remain flush-right
no matter how the window gets resized.
Your app might draw L/R itself (e.g. using GDI/OpenGL/DirectX inside the one window) or L/R might be some Microsoft control (which would have its own HWND separate from your main window HWND); doesn't matter.
Here is a simplified representation of the client area of your app window. As you can see, we have three-column-wide LLL at the far left of the client area, and three-column-wide RRR at the far right of the client area, with various other client area content represented by "-" in between (please ignore the grey background that StackOverflow insists on adding; L and R are at the far left and right edges of your client area):
LLL-----------RRR
Now imagine that you grab the left or right border of this window and drag it to make the window bigger or smaller.
1a. Easy Case: Drawing On Time
Imagine that your app is very fast at drawing so that it can always respond to the user's dragging action in 1 millisecond, and the OS lets your app draw that quickly without trying to draw anything else on the screen to "help" you.
As you drag the app border, the user sees the following on-screen (with each line of these figures representing one instant of time):
Dragging right border to the right (enlarging width):
(Figure 1a-1)
LLL-----------RRR (initially, when you click the mouse)
LLL------------RRR (as you drag the mouse)
LLL-------------RRR (as you drag the mouse)
LLL--------------RRR (when you release the mouse)
Dragging right border to the left (shrinking width):
(Figure 1a-2)
LLL-----------RRR
LLL----------RRR
LLL---------RRR
LLL--------RRR
Dragging left border to the left (enlarging width):
(Figure 1a-3)
LLL-----------RRR
LLL------------RRR
LLL-------------RRR
LLL--------------RRR
Dragging left border to the right (shrinking width):
(Figure 1a-4)
LLL-----------RRR
LLL----------RRR
LLL---------RRR
LLL--------RRR
These all look good and smooth:
When adjusting the right border, R appears to move at a constant speed in one direction and L stays still as it should.
When adjusting the left border, L appears to move at a constant speed in one direction and R stays still as it should.
So far so good.
1b. Hard Case: Drawing Falls Behind
Now, imagine that your app is so slow at drawing that the app cannot keep up with you as you drag with the mouse. Yes, eventually, your drawing will catch up, but we are talking about what happens during the time that you are dragging the mouse with your hand. Obviously the computer cannot reach out and grab your hand to slow your mouse movement down, so the key questions are:
what should show on the screen during this period, and
who decides what should show?
For example, when dragging the right border to the right (enlarging width):
(Figure 1b-1)
LLL-----------RRR
?????????????????? (what should show here?)
??????????????????? (what should show here?)
LLL--------------RRR (app catches up)
As another example, when dragging the left border to the left (shrinking width):
(Figure 1b-2)
LLL-----------RRR
???????????????? (what should show here?)
??????????????? (what should show here?)
LLL--------RRR (app catches up)
These turn out to be the key questions that determine whether the motion looks smooth or not, and they are the key questions around which this whole StackOverflow question revolves.
Different versions of Windows provide different answers to these questions in different contexts, meaning that the solution to getting smoother resize depends on which situation you are in.
1c. Temporary Solutions While Waiting for App to Draw
There are several choices of what to do in the period after the user has begun to drag the mouse to resize the window, but before your app has caught up by drawing the window at the new size.
1c1. Do Nothing
The screen could remain exactly as it is until the app catches up (neither your client pixels nor even the window border in the non-client area changes):
Example when dragging the right border to the right (enlarging width):
(Figure 1c1-1)
LLL-----------RRR
LLL-----------RRR
LLL-----------RRR
LLL--------------RRR (app catches up)
Example when dragging the left border to the left (shrinking width):
(Figure 1c1-2)
LLL-----------RRR
LLL-----------RRR
LLL-----------RRR
LLL--------RRR (app catches up)
The obvious disadvantage of this method is that during the period in question, the app appears to have "hung" and appears to be unresponsive to your mouse movements, because neither the R nor the '-' nor the L nor the the window border is moving.
Microsoft is often picked on for Windows being an unresponsive OS (and it's sometimes their fault and sometimes the fault of the app developer), so ever since Microsoft introduced live-resize (Windows XP?), Microsoft never uses the "do nothing" method by itself.
The "do nothing" method is annoying for the user and looks unprofessional, but it turns out (very non-obviously) that it's not always the worst choice. Read on...
1c2. Scale Content
Another possibility is that Windows could always make the window border follow your mouse movements instantly (because Windows itself has enough processing power to at least draw the non-client area in a timely manner), and while it is waiting for your app, Windows could take the old pixels of the client area and scale those pixels up or down just like when you zoom/blow up an image so that they "fit" in the smaller or bigger space.
This technique is generally worse than any other technique because it will result in a blurry image of your original content that is likely to be out of proportion. So nobody should ever do this in any case. Except, as we will see in PART 2, sometimes Microsoft does.
1c3. When Enlarging, Fill in Some Background Color
Another technique that could work when enlarging a window is the following: Windows could always make the window border follow your mouse movements instantly, and Windows could fill in new pixels of the now-larger client area with some temporary background color B:
For example, when dragging the right border to the right (enlarging width):
(Figure 1c3-1)
LLL-----------RRR
LLL-----------RRRB
LLL-----------RRRBB
LLL--------------RRR (app catches up)
This method has the advantage that during the period in question, at least your window border is moving, so the app feels responsive.
Another nice feature is that during the drag, L stays still, just like it should.
It's a little weird that the new space you are creating as you drag gets filled in with some random color, and even more weird that R doesn't actually move until later (notice that R jerks rightward by 3 columns at the last instant), but at least R only moves in the correct direction. It's a partial improvement.
A huge and important question is: what color should the newly filled-in background color B be? If B happens to be black and your app happens to have a mostly white background, or vice versa, it's going to be much uglier than if B matches your existing content's background color. As we will see in PART 2, Windows has deployed several different strategies to improve the choice of B.
Now consider the same idea, but instead apply it to the case where we are dragging the left border to the left (enlarging width).
The logical thing would be to fill in the new background color on the left side of the window:
(Figure 1c3-2)
LLL-----------RRR
BLLL-----------RRR
BBLLL-----------RRR
LLL--------------RRR (app catches up)
This would be logical because R would stay put, just like it should. L would have the same weirdness we described along with Figure 1c3-1 above (L would hold still and then jerk 3 columns leftward all of a sudden at the last instant), but at least L would only move in the correct direction.
However---and this is going to really come as a shock---in several important cases that you have to deal with, Windows does not do the logical thing.
Instead, Windows sometimes fills in background pixels B on the right even if you are dragging the left window border:
(Figure 1c3-3)
LLL-----------RRR
LLL-----------RRRB
LLL-----------RRRBB
LLL--------------RRR (app catches up)
Yes, this is insane.
Consider how this looks to the user:
L appears to move very smoothly at a constant speed in one direction, so that is actually good, but
Just look at what R is doing:
RRR
RRR
RRR
RRR (app catches up)
R first moves to the left by two columns, which it should not do: R is supposed to stay flush-right at all times
R then snaps back to the right again. Holy crap!
This looks horrible, terrible, abysmal, disgusting, ... there are not even words to describe how bad this looks.
The human eye is extremely sensitive to motion, even motion that happens over just a few frames of time. Our eye instantly picks up on this bizarre back-and-forth motion of R and we immediately know something is seriously wrong.
So here you can begin to get a sense of why some of these ugly resize problems only happen when you drag the left (or top) border and not the right (or bottom) border.
In reality, both cases (Figure 1c3-2 vs. Figure 1c3-3) do something weird. In Figure 1c3-2 we temporarily add some background pixels B that do not belong there. But this weird behavior is much less noticeable than the back-and-forth motion of Figure 1c3-3.
This back-and-forth motion is the jitter/flicker/jumping that so many StackOverflow questions are about.
So any solution to the problem of smooth resize has to:
at least prevent items in your client area from appearing to jump in one direction then back the other direction.
ideally also avoid the need to add background pixels B, if possible
1c4. When Shrinking, Cut Off Some Pixels
Section 1c3 dealt with expanding the window. If we look at shrinking the window, we will see there is an exactly analogous set of cases.
A technique that could work when shrinking a window is the following: Windows could always make the window border follow your mouse movements instantly, and Windows could simply chop off (crop) some pixels of your now-smaller client area.
For example, when dragging the right border to the left (shrinking width):
(Figure 1c4-1)
LLL-----------RRR
LLL-----------RR
LLL-----------R
LLL--------RRR (app catches up)
With this technique, L stays put as it should, but a weird thing happens on the right: R, which is supposed to stay flush-right no matter what the window size, appears to get its right edge incrementally sliced off by the right edge of the client area until R disappears, and then all of a sudden R reappears at its correct position when the app catches up. This is very weird, but keep in mind that at no point does R appear to be moving to the right. R's left edge appears to stay still, until the last moment when all of R jumps back 3 columns leftward. So, like we saw in Figure 1c3-1, R only moves in the correct direction.
Now consider what happens when we drag the left border to the right (shrinking width).
The logical thing to do would be to shave pixels off the left of the client area:
(Figure 1c4-2)
LLL-----------RRR
LL-----------RRR
L-----------RRR
LLL--------RRR (app catches up)
This would have the same weird properties as Figure 1c4-1, just with the roles of left and right reversed. L would appear to get incrementally shaved off from L's left edge but L's right edge would remain still until at the last instant L appears to jump to the right. So L only moves in the correct direction, albeit abruptly.
But---yes, get ready for total shock again---in several important cases that you have to deal with, Windows does not do the logical thing.
Instead, Windows sometimes chops pixels off of the right even if you are dragging the left window border:
(Figure 1c4-3)
LLL-----------RRR
LLL-----------RR
LLL-----------R
LLL--------RRR (app catches up)
Consider how this looks to the user:
L appears to move very smoothly at a constant speed in one direction, so that is actually good, but
Just look at what R is doing:
RRR
RR
R
RRR (app catches up)
R first slides over to the right by two columns. R's left edge appears to move rightward along with the rest of R.
R then snaps back to the left again.
As you should now be aware of after reading section 1c3, this back-and-forth motion looks absolutely horrible and is much worse than the admittedly weird behavior of Figure 1c4-1 and Figure 1c4-2.
1c5. Wait a Bit, Then Try One of Above
So far we have presented separate ideas for what to do when the user has begun to drag the window borders but the app hasn't redrawn yet.
These methods can actually be combined.
For a moment, try to think of this problem from Microsoft's point of view. At the moment that user starts dragging the mouse to resize your window, Microsoft has no way of knowing ahead of time how long it will take your app to draw. So Microsoft has to strike a balance:
if your app is going to respond quickly, then any changes Microsoft makes to the screen are going to make your app look worse than if Microsoft just lets you draw the real content (remember, all the tricks above are weird to varying degrees and will make your content appear strangely, so not using any of those tricks is definitely better).
but if Microsoft waits for you to draw for too long, your app (and Windows by extension) will look hangy and unresponsive as we explained in Section 1c1. This makes Microsoft lose face even if it's your fault.
So, another option is to first hold off on any screen changes and give the app a certain amount of time to draw, and if the app fails to meet the deadline, then employ one of the methods above to temporarily "fill in the gap."
Does this sound horrible and hacky to you? Guess what? That's what Windows does, in at least 2 different ways simultaneously with 2 different deadline times. PART 2 will dive into these cases...
PART 3: Gallery of Sorrow: Annotated List of Related Links
You might be able to glean ideas I missed by looking over the source material:
2014 with 2017 updates: Can't get rid of jitter while dragging the left border of a window : probably the most up-to-date question but still lacks context; suggests a creative but rather crazy hack of having two windows and alternately unhiding them during live resize! Also the only question I have found with an answer mentioning a race condition in DWM and a partial timing fix with DwmGetCompositionTimingInfo().
2014 Why is there a black lag every time a WPF window is resized? : yes WPF does it too. No useful answers
2009 How to fix the WPF form resize - controls lagging behind and black background? : controls lagging behind and black background?" multi-HWND example. mentions WM_ERASEBKGND and background brush tricks, but no modern answer.
2018 Is there a way to reduce or prevent form flickering when using WPF? : yes, still not fixed as of 2018.
2018 Reduce flickering when using SetWindowPos to change the left edge of a window
: unanswered question that got many obsolete recommendations like WM_NCCALCSIZE
2012 OpenGL flickering/damaged with window resize and DWM active : good statement of problem, answerers completely misunderstood the context and provided inapplicable answers.
2012 How to avoid transient updates in a GUI resize? : mentions the trick of intercepting WM_WINDOWPOSCHANGING and setting WINDOWPOS.flags |= SWP_NOCOPYBITS.
2016 Unity bug report: "Window resizing is very choppy and stutters (border does not smoothly follow the mouse)" typical bug report found in hundreds of apps that is partially due to the problem in this bug report, and partially due to certain apps having slow drawing. The only doc I EVER found which actually says that Windows 10 DWM clamps and extends the outer pixel of the old window, which I can confirm.
2014 Flickering on window when resizing from left side with pre-Windows-8 answer including CS_HREDRAW/CS_VREDRAW and WM_NCCALCSIZE.
2013 Resizing Window causes smearing near the right border with old-school Win-7-only solution to disable Aero.
2018 Flicker-free expansion (resize) of a window to the left an example of a multi-window (multi-HWND) case, no real answer.
2013 WinAPI C++: Reprogramming Window Resize : too ambiguously asked to tell whether it is about client-area flickering (like this question) or non-client-area flickering.
2018 GLFW bug "Resizing windows on Windows 10 shows jumpy behaviour" one of MANY such bugs which never explain the context, like many StackOverflow posts
2008 "Flicker Free Main Frame Resizing" CodeProject that actually does a StretchBlt but won't work in a Windows 8+ world, where app does not have control when incorrect pixels are shown on screen.
2014 Smooth window resizing in Windows (using Direct2D 1.1)? : Well-stated but unanswered issue with Windows 8+ DWM copy
2010 How do I force windows NOT to redraw anything in my dialog when the user is resizing my dialog? : WM_NCCALCSIZE fix to disable bitblt that no longer works in Windows 8+ since DWM corrupts the screen before app has chance to display.
2014 Flicker when moving/resizing window : roundup of previous fixes that do not work in Windows 8+.
2007 WinXP-era "reducing flicker" CodeProject recommending WM_ERASEBKGND+SWP_NOCOPYBITS
2008 early Google Bug report of new Vista DWM problems
Table of Contents
Because this is a complex, multi-faceted issue, I recommend reading the answers in this order:
PART 1: What Makes Resize Look Good or Bad?
PART 2: Identifying and Fixing Windows Resize Problems
2a: Resize Problems from SetWindowPos() BitBlt and Background Fill
2b: Resize Problems from DWM Composition Fill
2c: How to Diagnose Your Problem
as well as a list of source material which may help others glean insights:
PART 3: Gallery of Sorrow: Annotated List of Related Links
Please feel free to contribute more answers with creative ways of avoiding the problems described in 2a and especially 2b!
If you're using DXGI, you can use DirectComposition + WS_EX_NOREDIRECTIONBITMAP to bypass the redirection surface entirely and render/present the client area with the new size before even returning from WM_NCCALCSIZE (i.e. before any deadline timers even start). Here's a minimal example using D3D11:
#include <Windows.h>
#include <d3d11.h>
#include <dcomp.h>
#include <dxgi1_2.h>
ID3D11Device* d3d;
ID3D11DeviceContext* ctx;
IDXGISwapChain1* sc;
/// <summary>
/// Crash if hr != S_OK.
/// </summary>
void hr_check(HRESULT hr)
{
if (hr == S_OK) return;
while (true) __debugbreak();
}
/// <summary>
/// Passthrough (t) if truthy. Crash otherwise.
/// </summary>
template<class T> T win32_check(T t)
{
if (t) return t;
// Debuggers are better at displaying HRESULTs than the raw DWORD returned by GetLastError().
HRESULT hr = HRESULT_FROM_WIN32(GetLastError());
while (true) __debugbreak();
}
/// <summary>
/// Win32 message handler.
/// </summary>
LRESULT window_proc(HWND hwnd, UINT message, WPARAM wparam, LPARAM lparam)
{
switch (message)
{
case WM_CLOSE:
ExitProcess(0);
return 0;
case WM_NCCALCSIZE:
// Use the result of DefWindowProc's WM_NCCALCSIZE handler to get the upcoming client rect.
// Technically, when wparam is TRUE, lparam points to NCCALCSIZE_PARAMS, but its first
// member is a RECT with the same meaning as the one lparam points to when wparam is FALSE.
DefWindowProc(hwnd, message, wparam, lparam);
if (RECT* rect = (RECT*)lparam; rect->right > rect->left && rect->bottom > rect->top)
{
// A real app might want to compare these dimensions with the current swap chain
// dimensions and skip all this if they're unchanged.
UINT width = rect->right - rect->left;
UINT height = rect->bottom - rect->top;
hr_check(sc->ResizeBuffers(0, width, height, DXGI_FORMAT_UNKNOWN, 0));
// Do some minimal rendering to prove this works.
ID3D11Resource* buffer;
ID3D11RenderTargetView* rtv;
FLOAT color[] = { 0.0f, 0.2f, 0.4f, 1.0f };
hr_check(sc->GetBuffer(0, IID_PPV_ARGS(&buffer)));
hr_check(d3d->CreateRenderTargetView(buffer, NULL, &rtv));
ctx->ClearRenderTargetView(rtv, color);
buffer->Release();
rtv->Release();
// Discard outstanding queued presents and queue a frame with the new size ASAP.
hr_check(sc->Present(0, DXGI_PRESENT_RESTART));
// Wait for a vblank to really make sure our frame with the new size is ready before
// the window finishes resizing.
// TODO: Determine why this is necessary at all. Why isn't one Present() enough?
// TODO: Determine if there's a way to wait for vblank without calling Present().
// TODO: Determine if DO_NOT_SEQUENCE is safe to use with SWAP_EFFECT_FLIP_DISCARD.
hr_check(sc->Present(1, DXGI_PRESENT_DO_NOT_SEQUENCE));
}
// We're never preserving the client area so we always return 0.
return 0;
default:
return DefWindowProc(hwnd, message, wparam, lparam);
}
}
/// <summary>
/// The app entry point.
/// </summary>
int WinMain(HINSTANCE hinstance, HINSTANCE, LPSTR, int)
{
// Create the DXGI factory.
IDXGIFactory2* dxgi;
hr_check(CreateDXGIFactory1(IID_PPV_ARGS(&dxgi)));
// Create the D3D device.
hr_check(D3D11CreateDevice(
NULL, D3D_DRIVER_TYPE_HARDWARE, NULL, D3D11_CREATE_DEVICE_BGRA_SUPPORT,
NULL, 0, D3D11_SDK_VERSION, &d3d, NULL, &ctx));
// Create the swap chain.
DXGI_SWAP_CHAIN_DESC1 scd = {};
// Just use a minimal size for now. WM_NCCALCSIZE will resize when necessary.
scd.Width = 1;
scd.Height = 1;
scd.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
scd.SampleDesc.Count = 1;
scd.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT;
scd.BufferCount = 2;
// TODO: Determine if PRESENT_DO_NOT_SEQUENCE is safe to use with SWAP_EFFECT_FLIP_DISCARD.
scd.SwapEffect = DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL;
scd.AlphaMode = DXGI_ALPHA_MODE_IGNORE;
hr_check(dxgi->CreateSwapChainForComposition(d3d, &scd, NULL, &sc));
// Register the window class.
WNDCLASS wc = {};
wc.lpfnWndProc = window_proc;
wc.hInstance = hinstance;
wc.hCursor = win32_check(LoadCursor(NULL, IDC_ARROW));
wc.lpszClassName = TEXT("D3DWindow");
win32_check(RegisterClass(&wc));
// Create the window. We can use WS_EX_NOREDIRECTIONBITMAP
// since all our presentation is happening through DirectComposition.
HWND hwnd = win32_check(CreateWindowEx(
WS_EX_NOREDIRECTIONBITMAP, wc.lpszClassName, TEXT("D3D Window"),
WS_OVERLAPPEDWINDOW, CW_USEDEFAULT, 0, CW_USEDEFAULT, 0, NULL, NULL, hinstance, NULL));
// Bind our swap chain to the window.
// TODO: Determine what DCompositionCreateDevice(NULL, ...) actually does.
// I assume it creates a minimal IDCompositionDevice for use with D3D that can't actually
// do any adapter-specific resource allocations itself, but I'm yet to verify this.
IDCompositionDevice* dcomp;
IDCompositionTarget* target;
IDCompositionVisual* visual;
hr_check(DCompositionCreateDevice(NULL, IID_PPV_ARGS(&dcomp)));
hr_check(dcomp->CreateTargetForHwnd(hwnd, FALSE, &target));
hr_check(dcomp->CreateVisual(&visual));
hr_check(target->SetRoot(visual));
hr_check(visual->SetContent(sc));
hr_check(dcomp->Commit());
// Show the window and enter the message loop.
ShowWindow(hwnd, SW_SHOWNORMAL);
while (true)
{
MSG msg;
win32_check(GetMessage(&msg, NULL, 0, 0) > 0);
TranslateMessage(&msg);
DispatchMessage(&msg);
}
}
See the blog post The smooth resize test which has some analysis and pointers to solutions. Basically there is a winning strategy, which is to render to the redirection surface during live resizing, and use the swapchain at other times. I'm not sure if this fixes your specific problem, as you need enough low-level control of the way presentation works to be able to implement that. This approach also makes the assumption that you're drawing using Direct2D (as I'm currently doing) or DirectX.
Working on an application that controls a remote robot where there is the potential for significant delay between pressing a button and that action actually happening. Furthermore, there is the potential that the command did not successfully reach the intended recipient after all (due to network unreliability, etc.). Additionally, there are variables in play whose changes are not instantaneous. For instance, there is a variable both for commanded speed as well as current speed; changing the commanded speed will not immediately make the current speed match that value.
The question is, how do I make the application reflect both the current states the remote robot is reporting, as well as acknowledging to the user that his command was understood by the application, but the system has not yet received notification from the robot that it has been acknowledged? (Popups are an absolute no-go.)
Some ideas that have been discussed:
Disable Buttons
When a command button is pressed, start a timer for some reasonable number of seconds and disable the button during that time. Don't update corresponding label directly, but instead wait for a response from the robot. (e.g. if you press a Speed + button, and to the right is a text label showing current speed, don't immediately change the label but instead wait for a response from the robot). Once this response occurs, or when the timer expires, re-enable the button.
Pros: No additional control widgets needed on page. Labels always reflect current state of the robot.
Cons: If you wanted to send two speed updates in a row, would have to wait until first had been received and acknowledged. Would feel sluggish and unresponsive.
Logging Info
Have a log that users can view that shows a textual representations of all the actions the user has taken, timestamped, and with a history clearly visible. Could be color coded based on user preferences.
Pros: User has immediate feedback that his command was understood, as it appears in the log
Cons: Does not resolve problem of what to do with button (especially radio button) behavior.
Does anyone have experience with building UIs for environments in which there is significant latency between action and response? I would appreciate any and all input.
I would not go for a Log: your main focus is with the Widgets. There are several techniques for reporting status for different components, I will discuss a simple one here:
A button has an status-icon next to it show it's status. Use different colors to denote the latency. Green means "ready", when the user clicks the button, the icon changes to "orange" and that indicates busy. When the user clicks again, the color changes to "red", which means queued. When the queue is empty, the color changes back to orange. If the action was executed, the color is changed to green.
A slider can be used for floating values: use two "sliders". The first is better visible, and can be dragged. The second, and a layer below the first, is the "actual reported value", which shows the latency.
Textual input can also use the green/orange status-icons. While editing, the color changes to orange. If your queue/networking protocol supports canceling editing actions, you can resend the new string every time the user presses a key. If not: change the icon to orange upon a change, send the string, and wait for a status report. The status report should contain the actual value, and if this actual value is equal to the value in the component then change the icon to green. If the the actual is not equal to the value in the component then resend the value in the component.
Radiobuttons/Checkboxes should have a double-display. One editable, one uneditable. The first is for user-input and the second is for actual reported status. Same behavior as the slider component.
These require custom components, or widgets, to be made. You can extend original components or recreate them from scratch.
If your robot can also be "steered":
Create a rectangle which can be dragged upon. The rectangle has a small cross painted on, of the current value. While you drag, you see the latency of the cross. You can use interpolation and time values to smooth out robot control. A user will notice the lag because the cross "follows" his mouse-pointer. (Often used in space shooter simulation games when controlling virtual ships, Allegiance for instance)
It sounds like you have the following information that needs to be communicated to the user:
Current state or value of an attribute of the robot.
Target (i.e., received) value the robot is seeking.
Commanded value sent by the user to the robot.
Status of each command (pending, received, achieved, timed-out).
There also a couple other considerations:
Continuous or discrete feedback. Do you have continuous real-time feedback of the current value from robot? Or is it discrete feedback, where the robot sends the current value only after achieving a target? Obviously continuous is better for the user, since it allows the user to distinguish between the robot being slow and being stuck, but if you don’t have it, you have find a way to live without it.
Synchronous or asynchronous command-sending. If the commands are sent synchronously (i.e., no new one is sent out until the last one is known to be received), then the user may need a means to (a) force out the next command without waiting for a reply from the previous (in case the reply was lost), and (b) cancel a queued command in case conditions change between when a command was created and sent.
Robot with or without conflict resolution. Does the robot have the logic to look ahead in the list of commands it has received and resolved conflicting commands? For example, if it’s at a stop and its received-commands queue includes a command to go 5 m/s followed by a command to go 2 m/s, is it smart enough to delete the 5 m/s command? Or will it first attempt to accelerate to 5 m/s then 2 m/s, possibly resulting in an overshoot? Will it wait until it achieves 5 m/s before it even “looks at” the 2 m/s command? A lack of conflict resolution complicates your UI because the users may have to track all commands sent to understand why the robot is behaving like it is.
Integrated Information with Position-coding Controls
Let’s assume that the robot has conflict resolution and asynchronous command-sending. Rather than have separate controls for commanded, targeted and current values, I recommend you integrate them all in a single control to make it easy for the user to compare current, target, and commanded values, and see discrepancies. Perhaps the best way to do this is by representing values by positions in the window. Such position-coding of values is unmatched in showing the relations among things. There are two standard GUI controls, radio buttons and sliders, that accomplish such position coding. However, you’ll have to augment them to fulfill for your purposes. The best usability could require custom-made position-coding controls where you schematically represent the robot and maybe the environment, and allow the user to control it through direct manipulation. However, I'm assuming you need a simple-to-develop implementation, and a well-laid-out combination of sliders and radio buttons may get you pretty close to this ideal.
Use radio buttons for setting a categorical value and a slider for setting a numeric value. The slider may include a text box to allow the user to fine-tune the value. You augment each of these controls so that they show the commanded, target, and current values at the same time. The “usual” indicator (the dot for the radio button and the handle for the slider) represents the commanded value while separate graphic pointers indicates the current and target values. Discriminate the current from the target by making the current more prominent. I’d design them such that they merge into a single pointer when they are at the same value in order to minimize clutter for the usual state of things. If your users are untrained on the system you may want to include text labels on the pointers (“current,” and “target” when target is different than current).
Using these position-coding controls makes it easy for the user to compare current, target, and commanded values and see discrepancies. The status is implicit in the relative positions of the indicators. When the target pointer moves to the commanded position, the user knows the command was received. When current pointer is at the target, the robot has achieved the commanded value. This is especially good for continuous feedback of numeric values because the users can not only see the difference between the current and target value on a slider, they can estimate how long it will take for it achieve the target by seeing how fast the pointer is closing on the target. For the radio buttons, you can include “X% Done” text by the Target pointer to indicate when the target will be achieved (if this information is available).
For the most responsive UI, changing a value of a slider or radio buttons should send an immediate command. There is no “Apply” button. Users can re-send a commanded value at any time by re-clicking on the appropriate slider position or radio button. I think you'll find this is a natural human tendency anyway when confronted with an apparently unresponsive control (consider elevator buttons).
The descrepency between the target pointer and commanded indicator may be too subtle to signal a lack of reception of a commmand if responses are commonly slow (over a few seconds, such that user attention has likely shifted elsewhere). If that is the case, you may want to include a modeless alert after a time-out period that almost certainly indicates the command was lost in transmission. A modeless alert may include a text annunciator beside the control and/or graphically highlighting the commanded-target descrepancy. Depending on criticality, you may want to use a audible alert like a beep or animation to speed capturing user attention. The modeless alert disappears automatically when the target value matches the commanded value for whatever reason.
Separate Controls for Commanded and Current
If sliders and radio buttons take too much space for your purposes (or have other issues), you can go with separate non-position-coding controls for commanded and current values, as implied by your Disable Buttons design. However, overall, this is a more challenging design with more issues to resolve.
I would favor field controls like text boxes, check boxes, and dropdown menus, rather than command buttons so that the commanded value is clearly shown. Continuous numeric attributes may include spinner buttons with the text box if it doesn’t end up clogging the queue with incremental commands. As with the above option, changing a value sends an immediate command.
You’re right to be concerned about the using a timer and disabling. In addition to the problem of making the system sluggish, it means you gray out the commanded state. That can make it hard to read, and also requires some mental gymnastics by the user to interpret (“it’s unavailable, so that means I already selected it”). The interpretation can also be ambiguous because often disabled means Not Applicable (e.g., the Speed control is disabled because the robot has lowered stabilizers for fixed-base operation).
The solution is to use some other graphic feedback than disabling. I’d stay away from color coding. Color coding tends to be arbitrary and thus confusing (e.g., does red mean queued or timed-out?). This may be one of the (rare) good places to use animation since animation is intuitive for representing an on-going process. A flashing or throbbing border (or other feature) for the commanded-value control can indicate a sent command is awaiting reply. A flashing/throbbing border for the current-value control indicates the command is received and the robot is seeking the target value. If animation would be distracting in this situation (like it is for most other situations), then consider a dashed border (versus solid) to indicate awaiting reply or seeking target; dashed suggests a tentative or transitory state.
The target value and status are implied by what is animated. If the commanded-value border is animated, the value inside is a pending –the reply is yet to be received. If the current border alone is animated, the value inside the commanded control is the target value. If both borders are animated, then the robot is seeking one (unspecified) target value, while another is pending. If you think it’s problematic to leave the target unspecified in such circumstances, then you may need three controls to discriminate commanded, target, and current. However, if this is an edge case, it may be better to display the target value on mouse-over of current value control or with a drop-down button.
If feedback is continuous, you can also update the current value of numeric attributes at regular increments (about every 200 to 500 ms) so this animation provides an additional feedback of the robot seeking the target. For any attribute, if it takes 10-15 seconds or more for the robot to reach common targets and the robot has conflict resolution, you may want to also display a progress bar within or beside the current attribute control so the user can judge when the robot will achieve the target value.
To re-send the command, users can always re-select a value, or hit Enter while focus is on the commanded control. That’s a little odd and awkward for some controls (e.g., checkboxes), so I’d also consider a modeless notification (not a popup) that appear near the control if the command times out for a reply. The notification includes a button to resend the command.
If your users are untrained on the system you may want to include redundant text under the animation (e.g., “Sending” when waiting for a reply and “Seeking” when moving towards the target value).
Log Table
The logging approach is probably best if commands are sent synchronously and/or the robot lacks conflict resolution. This way the user can track the command queue for either sending commands or the commands received by the robot in order to predict robot behavior. However, I wouldn’t make it a read-only text box, but rather a table that can be manipulated. While the table is sorted by default by timestamp, there will be separate sortable fields for the attribute, the commanded value, the status (pending, seeking, achieved, timeout). If feedback is continuous, then the status should indicate progress towards achieving the target value (e.g., percent, or a progress bar).
If there is synchronous command-sending, then users can edit the commanded values of pending commands or force forward, move, or delete pending commands. In any case, commands can be copied and re-inserted in order to resend any command from any time. Maybe even provide a means to save selected commands and retrieve and insert them later –now you have macro facility.
If the robot tends to be is especially cranky (frequent loss of communications, slow responses), then you may want to have this log table beside the controls for creating commands and viewing the current values. The controls should be set up to make the creation of a discrete command clear to the user. With a cranky computer-to-robot interface, spurious commands are costly, so each command should be well-planned and deliberate. Likely this means a set of field controls like text boxes and drop-down lists to set values of various attributes and a button that generates the command(s) for those values. Awkward, yes, but that’s an accurate representation of the communication link with the robot.
Alternatively, if typically the queues are nearly empty, then you may want to make this table available in a separate window for experts to troubleshoot problems with robot behavior. Normally then the users use one of the other two options I gave above.
Maybe you can use a variation of the command pattern. Each action by the user generates a command, which goes into a queue. The queue is visible to the user in a printout on the screen. So you do not disable the button, but allow the user to press the button multiple times but show the user that the command is queued. At the same time you do not update the labels showing current state of robot until you receive the state from the robot.
In the queue the command could show its status somehow, maybe text and colour. And maybe you should allow the user to delete a command before it is processed by the robot(if that is possible).
So the queue might look like this:
Command Status Result of Action
speed+5 pending speed will increase to 200 (Delete This)
speed+5 pending speed will increase to 205 (Delete This)
speed-5 pending speed will decrease to 200 (Delete This)
and so on.
In the programming of a table-based application module (i.e. the user mostly enters tabular data in an already laid-out table), how would you reject user input for a given cell?
The scenario is: the user edits the cell, enters something (text, picture, ...) and you want them to notice when they finish editing (hitting enter, for example) that their entry is not valid for your given "format" (in the wider meaning: it can be that they entered a string instead of a number, that their entry is too long, too short, they include a picture while it's not acceptable, ...).
I can see two different things happening:
You can rather easily fit their entry into your format, and you do so, but you want them to notice it so they can change if your guess is not good enough (example: they entered "15.47" in a field that needs to be an integer, so your program makes it "15")
You cannot guess what to do with their entry, and want to inform them that it's not valid.
My question specifically is: what visual display can you offer to inform the user that his input is invalid? Is it preferable to refuse to leave the editing mode, or not?
The two things I can imagine are:
using colors (red background if invalid, yellow background for my case 1 above)
when you reject an input, do something like Apple does for password entry of user accounts: you make the cell "shaking" (i.e. oscillating left and right) for one second, and keep the focus/editing in their so they don't loose what they've typed.
Let's hear your suggestions.
PS: This question is, at least in my thought process, somehow a continuation and a specialization of my previous question on getting users to read error messages.
PPS: Made this community wiki, was that the right thing to do on this kind of question or not?
Be careful using autocorrection such as forcing user input to fit your format. See:
Is it acceptable to normalize text box content when it loses focus?.
It’s generally better to prevent invalid entries in the first place than to autocorrect them later. For example, if only integers are allowed, then you ignore any keying of the decimal point (along with all letters and most special characters). In some environments, you may want to provide a quiet audible signal that input is ignored (e.g., a dull thud).
As for when you need to alert the user to an error, how about a callout? Draw a bright line from the control or point in question (field, status annunciator, button, menu, location of a drag and drop) to the margin of the window and put a brief message (two or three words, like "Unrecognized date") in a balloon. Placing the message on the margin should keep it from occluding anything of interest in a crowded table.
The sudden appearance of the call-out should be sufficient to catch user attention, so it's okay to let the user move on to other cells in case they want to fix the error later. For efficiency, you may want to hold the user in the error-related field when the error originally occurs (since often the user wants to correct it right away), but then allow the next tab or mouse click to navigate the user away.
On mouseover or when focus is on the control associated with the error, the line is highlighted (to distinguish it from other callout lines that may be present) and the balloon expands to a full error message, providing more details on how to fix the problem (up to two sentences). Allow the user to drag and drop the balloon to a new location in case this occludes something of interest.
Include a Help button in the expanded balloon for further details. You can also include a button to fix the error (e.g., Retry, Reconnect, or set to default value).
The balloon disappears automatically when the error is fixed. Undo reverts whatever caused the error (e.g., reverts the field to its original value), which should clear the error.
If the user scrolls away from the place associated with the error, the balloon shrinks to an icon that remains in view so that the user is less likely to forget about it. Maybe an exclamation point in a triangle is a good icon. Place the icon beside or in the scrollbar track to indicate its relative location in the table, so the user can quickly scroll to find it later. Mouseover expands the icon to its full message. Perhaps clicking the icon can scroll to the right place in the table and put focus in the relevant control.
Balloons could also shrink to icons if they start visually interfering with each other. You may even want to include a control in a balloon to allow the user to force it to assume an icon.
For consistency, use for all errors, not just those associated with fields in tables.
The last time I did such a form (on a web page) I put a red box around the offending input.
I thought it was really neat... until a user asked me "Why's there a red box around this cell?"
What'd be nice is also displaying why the input's incorrect: "This field accepts only numbers", say.
You could display an icon in the cell, or a tooltip. The tooltip could open automatically or when mouse pointer hovers over the icon. It could disappear automatically when user edits another cell or when some timeout expires.
You can go with arrow tooltips like Adobe Flex's error tooltips. It focuses the attention to the error and supplies a brief description.
I have a table column where each row has one of three states, and the purpose of the row is to offer manipulation AND display of this property with three states.
My current development view is to have three tightly packed radio buttons with labels at the head of the columns (and every 50 rows or so) and onClick they send an AJAX request and thar she blows.
This is fugly.
Is there a standard idiom for a control like this? I'm currently mocking up something similar to the iPhone on/off toggle, but with a "middle" state.
Any input would be welcome.
EDIT
A bit more clarification: I have a tool for confirming events. Each event is either "proposed", "cancelled", or "confirmed". They all default to "proposed" until someone explicitly confirms or cancels them. This is a thin front-end for a SQL table.
I've seen this handled with image buttons that remain "depressed" when you click while popping the other two out. They act like radio buttons except that the label and the state are merged. If your names are too lengthy to fit in a button, you can abbreviate them and provide a key. I'd also give each one a distinct color. For implementation just pop the value in a hidden form field on click.
These are called "Toggle Buttons" in some other UI's:
http://java.sun.com/products/jlf/ed2/book/HIG.Controls2.html
http://msdn.microsoft.com/en-us/library/dd940509%28VS.85%29.aspx
http://developer.gnome.org/projects/gup/hig/2.0/controls-toggle-buttons.html
The standard mechanization for things like this in military avionics, where screen space is always at a premium,and so are buttons, is a "rotary". Each time you click it, it steps to the next value in sequence, wrapping around.
As an example, a device with a cryogenic cooler might have three states: OFF, WARM, and COOL. Initially, the device is OFF: no power applied. Click it, and it switches to WARM, meaning power is applied, but no cooling. Click it again, and it starts the cooler. (Since cooling in this kind of thing is usually supplied by a gas bottle with a strictly limited capacity, you don't want to cool the device until you are getting ready to party.) Click it again, and it shuts the device OFF.
You could also do this with buttons or hyperlinks. In a big table, hyperlinks will probably look best.
In the Proposed state, your cell could look something like this (with underlined links, but the editor won't let me):
Proposed Confirm Cancel
In the Confirmed state:
Confirmed Undo
In the Cancelled state:
Cancelled Undo
This will take two clicks to get from Confirmed to Cancelled and vice versa, but I assume that this operation is rarer than switching between Proposed and one of the other two.
Perhaps display arrows on either side to change the state:
(Cancelled) <| Proposed |> (Confirmed)
These may or may not 'wrap' depending on how well that suits the values and how important it is to saving a click when transitioning from value 1 to value 3 (or vice-versa).
As an alternative to you radio buttons, you could consider a drop-down list with three options. The disadvantage is, of course, that two clicks are needed to change the value.
Maybe use a slider with three states? (It really depends on the exact situation!)
Consider a fixed-position slider with three positions, such as offered by jQueryUI: http://jqueryui.com/demos/slider/#steps
I am reminded of the permissions button in SQL - it has multiple states; green check, red x, no setting, and clicking on them cycles through the three states. Its ok but annoying if you want to change a bunch to the state reached second, and if you click too many times you have to go through it all again. Left click - cycle forward; right click - cycle backward might work but certainly has no basis in UI expectations.
Idiomatically, I would say a Stop Light (red/yellow/green). They could behave like radio buttons; darker toned for 'off' and lighter tones for 'on', and since the color gives a cue you can move the description to a mouseover label. Of course, it isn't RG Colorblind kosher, so depending on your application that may be a deal breaker. (also, it may be confusing on Mac where the minimize/close etc buttons are the same color scheme).
Why not use three boxes that look like the "Questions", "Tags", "Users", ... boxes on this page (could be implemented as links, buttons or whatever)?