I'm looking for systematic review of the algorithms used for GUI layout. I'm particularly interested in the algorithms that favor speed over complexity, but it's hard to find anything useful other than "don't use constraint sovers".
So far the best one I've found is Constraint Solving for Beautiful User Interfaces: How Solving Strategies Support Layout Aesthetics by Clemens Zeidler,Christof Lutteroth and Gerald Weber. Unfortunately it is very short and is not very specific.
The layout algorithm landscape looks to me like this so far:
There's everybody's darling Cassowary that made its way just about everywhere.
There's the Auckland Layout Model which no one seems to use but thanks to its authors being prolific writers is well known.
There's the universally hated HTML/CSS combo that completely destroys the former two when it comes to the flow layout (especially the text flow).
Next come the algorithms used in desktop GUI frameworks, with WPF, Qt and MigLayout for Swing lauded the most. (I haven't figured out their performance characteristics yet, but I remember being disappointed by WPF layout speed compared to WinForms - the gains from faster drawing were offset by slower layout)
Than there's a handful of game engines UIs that no one seems to know anything about.
The rest are the ancient GUI toolkits that typically have an unsophisticated grid container as their most potent layout tool.
Have I missed anything significant?
Related
Years ago I discovered radial menus and realized that they could potentially be much easier to use than linear menus. At the time, I didn't have enough understanding of user experience principles (other than Fitt's law) to give them an honest evaluation. I thought that if they were indeed much easier to use in practice they should eventually show up in mainstream products, even against the inertia of the convention of linear menus.
Fast forward to today, I download ZGRViewer to visualize some graphs. It uses a radial context menu and I realize that these things really are easier to use, especially when you learn the simple mouse gestures to select nested menus.
Why hasn't the radial menu gained more traction? Is it user inertia? Framework inertia (i.e. not enough good implementations on various platforms)? Are they just not that much better in practice, overall?
In UX there are things we can do and some we would like to do but don't. Understanding how the human machine ingests information is critical in designing a great experience. A radial menu does make sense but it's not what a user has been accustom to. It's not impossible to implement but it will take the user some time to understand the context of where they were to where they are now. There are inherent behavioral patterns that can't be swept aside. You learn to push the boundaries and still make it usable just beyond the visual appeal.
We have a number of applications that are now looking tired and a bit drab. Looking at the MS style fluent interface looks nice but seems (to me) to be more document based rather than task based.
Is there a nice 'modern' ui style that lends itself to task based applications?
Graphic style is pretty much independent of whether a UI is object-centered or task-centered. If you’re just looking to refresh the appearance, I don’t see why you can’t take some of the elements from MS’s examples and put them in your apps (e.g., various shades of one color, gradients and swoops, “soft” abstract icons, rounded corners). Check out MS’s own Inductive User Interface Guidelines for screen-shots of task-based apps that use some of these elements.
Alternatively, you could try to be original, rather than merely fashionable. Just a thought.
While you’re at it, take this opportunity to fix some usability problems or other functional weaknesses you’ve been observing over the years. If you don't give your users something practically better, they're going to associate the new style with making life difficult, rather than something exciting.
Consider a single button.
At one extreme, we have a black OpenGL window, with:
outline (in white) of a rectangle
bitmap remdered font inside of it, saying "Ok"
At the other extreme, we have Mac OS X, a button that is:
well rounded
has some gradient showing light effects on it
nice antialiased "OK"
soft shadow of some sort
These two UIs present very very different user experiences. The former says "This is from the 80s" the latter says "this is professional".
This is something I do not understand well as a programmer (and don't know where to learn about this).
Does anyone know of a good technical resource for this? [I'd prefer things that draws upon psychology / perception literature to say why to do something rather than design books that just says "use color XYZ with a gradient of blah"]
Here is something on it. http://www.alistapart.com/articles/indefenseofeyecandy
You can check this link out to answer the part of your query in the comment. It has lots of references to samples and some helpful links too. http://www.usernomics.com/user-interface-design.html
The perception and psychology part of designing the UI does not come as any rule or steps, as we all know. It gets developed over time. Making your application user friendly and pleasing, that part of the magic or deal gets added from experience\survey and also you can include layman testing. I do it many times.
Also thinking out of the box. You will get a solution when solve it within the box. But you will get a better solution when you think out of it.
Another useful thing is be a good learner and observer. Note something nice and useful when you visit sites or use other applications. You might not even notice it. It might be something very small or trivial but it makes a lot of difference when it's used in the right places.
You will want to read up on Human User Interface guidelines HIG and Usability:
Apple's Human Interface Guidelines
Windows User Experience Guidelines
Platform agnostic guidelines
Amazon has plenty of books on the HIG subject, but I'd also recommend books based on usability. Steve Krug's "Don't make me think" is a great book (mainly tailored for web usability)
etc.
A classic: The Design of Everyday Things
Pretty quick read, discussed some of the psychology behind using and understanding human interfaces. It's a bit dated and doesn't directly focus on programming GUIs but I would start here.
I'd start with Vitruvius' firmness, commodity, and delight.
Also Gibson's affordances. Also, many HCI researchers have applied activity theory, with mixed results. Norman's DOET is a good start, but I think it covers only the first 2 of Vitruvius' triad - you're asking about the delight. Might also look at McCloud's Understanding Comics.
The Opera web standards curriculum has a very good section on aesthetic aspects, especially regarding color usage. I think it's very useful reading not only for web development, but all application design.
Chapter 8: Color Theory
Chapter 9: Building up a site wireframe
There is also a color scheme designer website, which allows you to play around with some of the color theory aspects. Definitely worth a visit.
I am giving a short talk on the subject from a programmer's perspective. I was thinking about talking about UI design along 3 axes:
Proximity: How close the code you write is to the UI
Interactivity: How flexible and powerful the UI can be
Complexity: How difficult it is to build a user interface (technical/implementation)
I was then going to discuss the following UIs:
Command Line:
High on proximity, somewhat high on complexity, low on interactivity
VB / MFC / Windows Forms
Low on proximity (deal with a big API/controls/UI Model), medium interactivity - you can build it in but with a high penalty in proximity (3rd party libraries/controls), bigger range on complexity; simple applications are easy to build.
Web
Low on proximity (browser does the legwork, different browsers behave differently), high on interactivity (DOM/CSS/Javascript/Browser), high on complexity (lots of moving parts)
Silverlight (perhaps other RIAs)
Higher proximity than RAD/MFC/VB style interfaces as well as Web, high on interactivity, lower on complexity (e.g. animation in this environment versus OpenGL etc)
Questions
Do these 3 axes make sense? I know there are some generalizations one could make such as a tradeoff of interactivity and complexity that apply in every environment but I see strengths and weaknesses inherent to each.
Is there another axis or trait in UIs that makes sense to discuss?
Future implications on how we develop (RIAs become dominant, etc...)
Any other feedback (+/-)
I think the key to your approach is to know your audience, are they other developers, or people who would like to know more about the development process?
I think the absolute key to UI is usability, everything else (including ease of development) is subordinate to that. Progammability is normally a result of tools, and if the technology is pervasive enough then the tools will come, windows dev used to be awful, and then a few years later everyone was a VB coder.
I think portability also is worth mentioning. Quite often, the UI components are what prevents the application from being ported to other platforms.
For the interactivity axis, I think usability needs to be discussed because flexibility and power are only helpful as they make a UI more usable. A user needs to rapidly be able to determine how to use the software, it ought to be intuitive.
All you Stackoverflowers,
I was wondering why GUI code is responsible for sucking away many, many cpu cycles. In principle, the graphical rendering is far less complex than Doom (although most corporate GUIs will introduce lots of window dressing). The event handling layer is also seemingly a heavy cost, however, it seems that a well-written implementation should switch between contexts efficiently on modern processors with a lot of memory/cache.
If anybody has run a profiler on their big GUI application, or a common API itself, I'm interested in where the bottlenecks lie.
Possible explanations (that I imagine) may be:
High levels of abstraction between hardware and application interface
Lots of levels of indirection to the correct code to execute
Low priority (compared to other processes)
Misbehaving applications flooding API with calls
Excessive object orientation?
Complete poor design choices in API (not just issues, but design philosophy)
Some GUI frameworks are much better than others, so I'd like to hear varied perspectives. For example, the Unix/X11 system is much different than Windows and even than WinForms.
Edit: Now a community wiki - go for it. I have one more thing to add -- I'm an algorithms guy in school and would be interested if there are inefficient algorithms in GUI code and which they are. Then again, it's probably just the implementation overhead.
I've no idea generally, but I'd like to add another item to your list - font rendering and calculations. Finding vector glyphs in a font and converting them to bitmap representations with anti-aliasing is no small task. And often it needs to be done twice - first to calculate the width/height of the text for positioning, and then actually drawing the text at the right coordinates.
Also, most drawing code today relies on clipping mechanisms to update just a part of the GUI. So, if just one part needs to be redrawn, the code actually redraws the whole window behind the scenes, and then takes just the needed part to actually update.
Added:
In the comments I found this:
I'm also very interested in this. It can't be that the gui is rendered using only the cpu because if you don't have proper drivers for your gfx-card, desktop graphics render incredibly slow. If you have gfx-drivers however desktop-gfx go kinda fast but never as fast as a directx/opengl app.
Here's the deal as I understand it: every graphic card out there today supports a generic interface for drawing. I'm not sure if it's called "VESA", "SVGA", or if those are just old names from the past. Anyway, this interface involves doing everything through interrupts. For every pixel there is an interrupt call. Or something like that. The proper VGA driver however is able to take advantage of DMA and other enhancements that make the whole process WAY less CPU-intensive.
Added 2: Ah, and for OpenGL/DirectX - that's another feature of today's graphics cards. They are optimized for 3D operations in exclusive mode. That's why the speed. The normal GUI just utilizes basic 2D drawing procedures. So it gets to send the contents of the whole screen every time it wants an update. 3D applications however send a bunch of textures and triangle definitions to the VRAM (video-RAM) and then just reuse them for drawing. They just say something like "take the triangle set #38 with the texture set #25 and draw them". All these things are cached in the VRAM so this is again way faster.
I'm not sure, but I would suspect that the modern 3D-accelerated GUIs (Vista Aero, compiz on Linux, etc.) also might take advantage of this. They could send common bitmaps to the VGA up front and then just reuse them directly from the VRAM. Any application-drawn surfaces however would still need to be sent directly every time for updates.
Added 3: More ideas. :) The modern GUI's for Windows, Linux, etc. are widget-oriented (that's control-oriented for Windows speakers). The problem with this is that each widget has its own drawing code and associated drawing surface (more or less). When the window needs to get redrawn, it calls the drawing code for all its child-widgets, who in turn call the drawing code for their child-widgets, etc.. Every widget redraws its whole surface, even though some of it is obscured by other widgets. With above mentioned clipping techniques some of this drawn information is immediately discarded to reduce flickering and other artifacts. But still it's lots of manual drawing code that includes bitmap blitting, stretching, skewing, drawing lines, text, flood-filling, etc.. And all this gets translated to a series of putpixel calls that get filtered through clipping filters/masks and other stuff. Ah, yes, and alpha blending has also become popular today for nice effects which means even more work. So... yes, you could say this is because of lots of abstraction and indirection. But... could you really do it any better? I don't think so. Only 3D techniques might help, because they take advantage of GPU for alpha-calculations and clipping.
Let's begin by saying that writing libraries is much harder than writing a stand-alone code. The requirement that your abstraction be reusable in as many contexts as possible, including contexts which you haven't though of yet, makes the task challenging even for experienced programmers.
Amongst libraries, writing a GUI toolkit library is a famously difficult problem. This is because the programs which use GUI libraries range over a very wide variety of domains with very different needs. Mr Why and Martin DeMollo discussed the requirements placed of GUI libraries a little while ago.
Writing GUI widgets themselves is difficult because computer users are very sensitive minute details of the behavior of the interface. Non-native widget never feel right, don't they? In order to get non-native widget right -- in order to get any widget right, in fact -- you need to spend an inordinate amount of time tweaking the details of the behavior.
So, GUI are slow because of the inefficiencies introduced by the abstraction mechanisms used to create highly-reusable components, that added to shortness of time available to optimize the code once so much time has been spent just getting the behavior right.
Uhm, that's quite a lot.
The most simple but probably obvious answer is that the programmers behind these GUI apps, are really bad programmers. You can go along way in writing code which does the most bizarre things and it will be faster but few people seem to care how to do this or they deem it to be an expensive non-profitable time wasted effort.
To set things straight off-loading computations to the GPU won't necessarily fix any problems. The GPU is just like the CPU except it's less general purpose and more a data paralleled processor. It can do graphics computations exceptionally well. Whatever graphics API/OS and driver combination you have doesn't really matter that much... well OK, with Vista as an example, they changed the desktop composition engine. This engine is far better composting only that which has changed, and since the number one bottle neck for GUI apps is redrawing is a neat optimization strategy. This idea of virtualizing your computational needs and only update the smallest change every time.
Win32 sends WM_PAINT messages to windows when they need to be redrawn, this can be a result of windows occluding each other. However it's up to the window itself to figure out whats actually changed. More than so nothing did change or the change that was made was trivial enough so that it could have been just preformed on top of what ever top most surface you had.
This kind of graphics handling doesn't necessarily exist today. I would say that people have refrained from writing really efficient and virtualizing rendering solutions because the benefit/cost ration is rather low/high (bad).
Something Windows Presentation Foundation (WPF) does, which I think is far superior to most other GUI API is that it splits layout updates and rendering updates into two separate passes. And while WPF is managed code the rendering engine is not. What happens with rendering is that the managed WPF rendering engine builds a command queue (this is what DirectX and OpenGL does) which is then handed of to the native rendering engine. What's a bit more elegant here is that WPF will then try to retain any computation which didn't change the visual state. A trick if you may, where you avoid costly rendering calls for things that doesn't have to be rendered (virtualizing).
In contrast to WM_PAINT which tells a Win32 window to repaint itself a WPF app would check what parts of that window requires repainting and only repaint the smallest change.
Now WPF is not supreme, it's a solid effort from Microsoft but it's not the holy grail yet... the code which runs the pipeline could still be improved and the memory footprint of any managed app is still more than I would want. But I hope this is the kind of answer you are looking for.
WPF is able to do some things asynchronously rather decent, which is a huge deal if you wanna make a really responsive low-latency/low-cpu UI. Asynchronous operations is more than off-loading work on a different thread.
To summarize things slow and expensive GUI means too much repainting and the kind of repainting which is very expensive i.e. the entire surface area.
I does to some degree depend on the language. You might have noticed that Java and RealBasic applications are a fair bit slower than their C-based (C++, C#, Objective-C) counterparts.
However GUI applications are much more complex than command line apps. The Terminal window needs only to draw a simple window that doesn't support buttons.
There are also multiple loops for extra inputs and features.
I think that you can find some interesting thoughts on this topic in "Window System Design: If I had it to do over again in 2002" by James Gosling (the Java guy, also known for his work on pre-X11 windowing systems). Available online here[pdf].
The article focuses on the positive side (how to make it fast), not on the negative side (what's making it slow), but it is still a good read on the topic.