Related
For about a year a half, I've been working with SilkTest, which is a GUI automation tool, for both desktop and web applications. It simulates mouse and keyboard inputs, which eventually simulate end user behaviour. However, I find that it is a bit flaky; Button.Click() or DialogBox.Close() method calls that work just fine 9 times in a row seem to fail on a 10th call, only to go back working on the 11th. Normally I would just chalk this up to a quirk with SilkTest (or the application under Test, or the OS, or what have you) but then I see that there are similar issues with other GUI automation tools like Selenium:
Selenium Click() fails with Anchor Elements
Selenium Click() fails clicking button object
I know that for desktop apps, each GUI control/dialog has a tag element associated with it (at least in Windows-based GUIs) and that for web pages there is the domain object model hierarchy of page elements. My guess is that these tools sometimes run into issues navigating these hierarchies and finding unique elements and controls. But what is going on here? SilkTest is a relatively old, commercial software package while selenium is relatively new, open source and constantly evolving. The fact that they both can have similar problems raises a couple of flags with me.
Also, is this the case with other GUI test tools? Or have I just had a somewhat unusual experience?
There are 2 things here that you are talking about, first the concept of finding an object in the application under test that you want to automate. Your description of how SilkTest (and other tools) does this is quite accurate, i.e. as long as there is something that the automation software can use to identify the control then you are fine.
The second thing is why does the automation itself fails randomly, since the tool has not reported that it could not find the control then it must think that it sent the appropriate action to the application, e.g. a Click or a Type. This could be that the application is not ready to accept the action that you are sending it, this is similar to you attempting to click on something "before it was ready", in this case the application can decide to buffer the input or to discard the input.
So, how do you fix this? One way would be to use the capabilities of the tool to try to work out when the application is ready for input rather than sending it a stream of input blindly. SilkTest has capabilities that allow for you to do this (as does TestPartner). I cannot comment on Selenium as it is something I have not used.
A simple way of testing this would be to insert a pause for a couple of seconds before the offending action, then run this in a loop to see whether this solves the problem, if this is the case then it is your problem. If this does not fix the issue then there is something else going on that you need to contact the vendor of the testing tool.
Remember that applications are getting more and more complex, i.e. multi-threading, communications, any one of these could cause the automatic syncronisation to fail causing actions to fail.
Hope that helps.
A quick question. Is there any method to control or automate any Windows application, using the command line. I've tried AutoIt. Any other methods? I'm targetting to control WinCE Test Kit (CETK) to perform the test without having to go to the GUI,or click the menu, connect etc, manually.
Thanks in advance!
We use Rational Robot for this but keep in mind it's not cheap. It's also probably been renamed 27 times since we started using it so you may want to just search for Rational testing products in general.
It's fully script-able, allowing you to monitor the screen and send key presses and whatnot.
Look at SCAR, it is great for all sorts of automation and screen reading: http://freddy1990.com/index.php?page=product&name=scar
I've always used a hybrid approach. First purchase SOTI Pocket Controller Pro then just use the normal AutoIt automation tools. It's a little different because you can't actually capture popup windows like you may be used to, but it can automate clicks, and then loop and wait for things using the GetPixel methods to check it the screen is what you expect.
Being able to connect to multiple CE devices at the same time visually is also a nice touch.
Is it worth to try to keep your GUI within the system looks ?
Every major program have their own anyways...
(visual studio, iexplorer, firefox, symantec utilities, adobe ...)
Or just the frame and dialogs should be left in the system look 'n feel range ?
update:
One easy exemple, if you want to add a close button to your tab, usually you make it against your current desktop theme. But if the user has a different theme, your close button is out of place, it doesn't fit the system look anymore.
I played with the uxtheme api, but there is nothing much you can do, and some themes i've seen are incomplete sets.
So to address this issue, the best way i see, is to do like visual studio/firefox/chrome roolup your own tab control with your theme...
I think, that unless your program becomes a very major part of the users life, you should strive to minimize "surprises" and maximimze recognizability (is that even a word?).
So, if you are making something that is used by 1.000 people for 10 minutes a day, go with system looks, and mechanisms.
If, on the other hand, you are making something that 100 people are using for 6 hours a day, I would start exploring what UI improvements and shortcuts I could cram in to make those 6 hours easier to deal with.
Notice however, that UI fixes must not come at the expense of performance. This is almost always the case in the beginning when someone thinks that simply overriding the OnPaint event in .Net will be sufficient.
Before you know it you are once again intercepting NC_PAINT and NC_BACKGROUNDERASE and all those little tricks to make it go as fast as the built-in controls.
I tend to agree with others here- especially Soraz and Smaci.
One thing I'll add, though. If you do feel that the OS L&F is too constraining, and you have good grounds for going beyond it, I'd strive to follow the priciple of "Pacing and leading" (which I'm borrowing here from an NLP context).
The idea is that you still want to capitalise as much as possible on your intended audidences familiarity with the host OS (there will be rare exceptions to this, as Smaci has already covered). So you use as much as possible of the "standard" controls and behaviours (this is the "pacing") - but extend it where necessary in ways that still "fit in" as much as possible (leading).
You've already mentioned some good examples of this principle at work - Visual Studio, even Office to some extend (Office is "special" as new UI styles that cut their teeth here often find their way back into future OS versions - or de-facto standards).
I'm bringing this up to contrast the type of apps that just "do it their way" - usually because they've been ported from another platform, or have been written to be cross-platform in GUI as well as core. Java apps often fall into this category, but they're not the only ones. It's not as bad as it used to be, but even today most pro audio apps have mongrel UIs, showing their lineage as they have been ported from one platform to another through the years. While there might be good business reasons for these examples, it remains that their UIs tend to suck and going this route should be avoided if in any way possible!
The overriding principle is still to follow the path of least surprise, and take account of your user's familiarity with the OS, and ratio of their time using your app to others on the OS.
Yes, if only because it enables the OS to use any accessability features that are built in like text-to-speech. There is nothing more annoying for someone who needs accessability features to have yet another UI that breaks all the tools they are used to.
I'd say it depends on the users, the application and the platform. The interface should be intuitive to the users, which is only the same as following system UI standards if they are appropriate for those users. For example, in the past I have been involved in developing hand held systems for dairy and bread delivery on Windows CE hand helds. The users in this case typically were not computer literate, and had a weak educational backround. The user interface focussed on ease of use through simple language and was modelled on a pre-existing paper form system. It made no attempt to follow the Windows look and feel as this would not have been appropriate.
Currently, I develop very graphical software for a user group that is typically 3rd level educated and very computer literate. The expectation here is that the software will adhere to and extend the Windows look and feel.
Software should be easy and intuitive where possible, and how to achieve this is entirely context dependent.
I'd like to reply with another question (Not really Stackoverflow protocol, but I think that, in this case, it's justified)
The question is 'Is it worth breaking the OS look and feel?'
In other words,
Do you have justification for doing so? (In order to present data in some way that's not possible within normal L&F)
What do you gain from doing so? (Improvinging usability?)
What do you lose from doing so? (Intuitiveness & familiarity?)
Don't simply do it 'To be different'
It depends on how wide you would define system look'n feel... But in general, you should keep it.
Do not surprise the user with differentiating from what he is used to. That's one of the reasons why we call him user ;-)
Firefox and Adobe products usually don't because they are targeting several plattforms which all have their own L&F. But Visual Studio keeps the typical Windows L&F. And, as long as you are developing only for Windows, so should you.
Apart from the fact that there is no well-defined look-n-feel on Windows, you should always try to follow the host platform native L&F. Note however that look-n-feel is just as much about how a program behaves as how it looks. Programs which behave in a counter-intuitive way is just as annoying as programs sporting their own ugly widgets.
Fraps is a good example (IMHO) of a program which is actually very useful, but breaks several user interface guidelines and looks really ugly.
If you're developing for Apple's Mac OS X or Microsoft Windows, the vendors supply interface guidelines which should be followed for any application to be "native".
See Are there any standards to follow in determining where to place menu items? for more information.
If you are on (or develop for) a Mac, then definitely YES!
And this should be true for Windows also.
In general, yes. But there's the occassional program that does well despite being not formatted for all the OSes it runs on. For example, emacs runs pretty much contrary to every interface guideline on OS X or Windows (and probably even gnome/KDE) and it's not going away any time soon.
I strongly recommend making your application look native.
A common mistake that developers who are porting an application to a new platform seem to make is that the new application should look-and-feel like it does on the old platform.
No, the new application should look-and-feel like all the other application that the user is used to on the new platform.
Otherwise, you get abominations like iTunes on Windows. The same UI design may be exactly right on one platform and very wrong on the next.
You will find that your users may not be able to pin-point why they dislike your application, but they just feel it hard to use.
Yes, there are valid exceptions, but they are rare (and sure enough, they tend to be the major applications like Office and Firefox, rather than the little ones). If you are unsure enough to have to ask on StackOverflow, your application isn't one of them.
We've got products built both with GUI and CHUI. Going forward, we're looking at redesigning a lot of our software and mainly taking the route of going all GUI. My question to the group is, do we need to account for keeping a CHUI around? What are the advantages of CHUI over GUI? Many times in the past people have said that CHUI is faster because you don't need a mouse. I argue that GUI can be just as fast with the right keyboard shortcuts, hotkeys and/or touch screens.
Is CHUI something we should no longer consider if hardware no longer provides a constraint?
Also to clarify, when I speak about CHUI I mean a CHaracter based User Interface, and I'm also mainly concerned with the effective presentation of data to an end user.
There have been some fantastic responses that have highlighted the importance of having a command line based interface for automation and scripting based tasks which I will certainly take to heart when we begin the design!
The primary benefits of a CHUI (that is something with forms and fields, not necessarily command line interfaces) is the keyboard for navigation and consistent layout. That is key.
If your GUI can be completely, and efficiently, keyboard navigated, then your CHUI user base should be happy. This is because in time, the users simply "type" their commands in to the system without "seeing the interface". They don't need to "discover" the interface, which is a primary feature of the GUI.
While CHUIs appear to be dinosaurs, they are still functional and usable. Most folks once they're trained (notably POS/Counter workers, but even back office scenarios like factory or warehouse floor, etc) have no problem using a CHUI.
But the key is the keyboard support so the user don't have to wait for the screen to catch up with them. Seeing a skilled operator with a mastery of the keyboard can make an application fly. You barely have a chance to see popup windows and what not.
You should poll your customers, not programmers. If your customers, who use your applications, want a CHUI, even if all your developers think it's a waste of time, you build it, because the customer is always right (except for when they're wrong).
You should absolutely still consider it. Most importantly, command line programs can be automated (and chained together in scripts) much more easily than GUIs (typically). I can't imagine working with a source control tool which didn't have a command line interface - although obviously having a GUI is useful too.
Now whether you need a command line version for your particular app is hard to say without knowing what your app does. Do you need automation and scripting? Might someone want to VPN in and run it from a very bad connection, and thus appreciate low bandwidth?
Note that MS certainly doesn't believe the command line is dead - or they wouldn't have created PowerShell.
I agree with Eli that your customers should have final say, but if you can keep the meat of your program from being too interwoven with the GUI(or CHUI), then production cost to make both available should be minimal.
If you write apps for unix and you need to handle users who telnet / ssh to your box then you will need command line interfaces.
I would say it depends on your target. Do you script your code from other apps? That would be a requirement to keep the interactive version (or some piece to avoid the GUI startup).
We usually do one or the other. But sometimes we have utils that have to be deployable through ftp and run ssh. Or we have tools that our users embed into their apps and don't want to expose a UI (data migration / conversion).
To this day, some of the most efficient user interfaces I've ever seen were plain old terminal-based character interfaces.
Anecdote: I was once part of a project to "modernize" a terminal application used by 500 customer service representatives. We published sexy GUI mockups and everyone, including the users, were suitably impressed. We worked for six months on the application, and all the user acceptance testing seemed to indicate we had a winner.
But when the application was finally launched, it failed miserably. As it turns out, CSRs are measured for performance daily, right down to the average number of seconds per call handled. And no matter how hard they tried, they could not match the same level of efficiency in the GUI as they could in the terminal interface. They could get close with tabs and shortcuts, but not quite there.
Hard lessons learned. Modern programmers may abhor "dinosaurs", but do users really care about slick interfaces? Usually they just want to get their work done.
When I first read this, my immediate thought was that this is probably one of those apps that's basically a series of forms, but displays inside a terminal. Often you see such dinosaurs running on cash registers. I also recall seeing such an app used to apply for a loan when I bought my car. This type of application doesn't seem to have a place in the modern world -- any system with even a tiny bit of processing power can handle a normal GUI nowadays. Unless you're trying to support really low-end legacy customers, get rid of this user interface. A GUI with decent keyboard shortcuts (please, please, please put some thought into keyboard-only use of your GUI programs...) is going to be equally effective for the users coming from the old CHUI system and much friendlier to those used to a GUI, without having to have 2 versions of your app.
I don't see why everyone is bringing up command line apps. I think most people recognize that the command line isn't going away. It's far faster for many tasks than a GUI, largely because the programs tend to be non-interactive (and thus easily scriptable). As soon as your app becomes interactive (or, at least, doesn't have a param to make it non-interactive), running it from the command line is much less important. Even awesome programs like Vim that are terminal-based are transitioning to their graphical counterparts (gVim) because it gives you the best of both worlds.
Even GUI apps like Firefox can benefit from command line interfaces like Ubiquity. If there's a way to provide the command line from within the GUI then why not have the best of both worlds?
A lot of CAD programs have command line interfaces that show you what the GUI interaction you just performed equates to in the command line. That way you can learn the command line operations for the things that you do frequently and where the command line can be quicker to interact with whist still having the discoverability of the GUI interface.
See this youtube video demonstrating Rhino3D's command line
CHUI is faster in execution speed, not user interaction speed. I write embedded systems (as well as GUIs), so I'll always have a use for command line apps.
Every study I have ever read showed that CHUI's are much faster for experienced users. GUI's are easier for new users and for applications that are only occasionally used. Also for a given screen size, you can display more information on a CHUI then a GUI. A good GUI can give you a quick over view at a glance.
In addition to the other benefits mentioned above, I've frequently found another reason to keep around an alternative UI--it keeps you and your interfaces honest. When an application is built with only one user interface, it becomes much easier to let design principles slide and for your business logic, etc. and your GUI to become an intertwined ball of spaghetti--despite best intentions. Regardless of the importance of your customers having a command-line interface, soon there might come a time when an alternative GUI (read: presentation layer) might be needed, and you'll want to be prepared. This might not be relevant to your requirements, but I think it's something good to keep in mind...
One of the big issues that we encountered was multisession capability which is almost nonexistent with the GUI technologies I have seen. Our users were quick to point out that with the current character based interface they could have over a dozen Telnet based terminal sessions going at the same time on their PC screen which enabled them to multitask or task switch with high efficiency. They rated multitasking as the killer feature which they benefitted from in our fast paced environment where interruptions are frequent. Being able to have concurrent access to multiple instances of a particular ERP application or multiple different ERP applications while always retaining session states was important to our user community.
I think the problem comes from design practices in GUI forms. We tend to place more objects on them especially with a vertical scroll bar and tab capabilities. This also makes loading slower. Going through CHUI menus with the keyboard is faster once you've memorized those sequences and holding the Ctrl key isn't required. There is something about the menu bar in Windows where the short-cut key descriptions are off to the right. The character based menus seemed easier to remember after awhile.
A) - This Menu
B) - That Menu
C) - Some other Menu
Or you could arrow through the choices and you just seemed to have some muscle memory where That Menu is the second choice.
As soon as you present some data, someone's going to want to query against it. You can integrate that with a gui, no problem. If you think some of your customers are going to want to script certain tasks. set it up. Anything to do with automation is better done from the command line(y harlo thar cron job!)
I love guis. I'm a mac user. But there is a time and a place for a CLI.
I was sysadmin at a university math department when the registration system went from a character based system using telnet, to a gui system on a PeopleSoft app.
The gals in the front office HATED the new system. Now part of this was the whole bit about old shoes being more comfortable. But when I asked about it, Christine said that even after a week of doing several hundred registrations per day, the new system took several times as long to do anything. Lots of things only doable with a mouse. The old system could accept input as fast as they could type. Screen repaints were under a tenth of a second. New system had lots of 3/4 to 2 second pauses -- just long enough to be annoying, not long enough to do anything else.
Does anyone have experience with embedding messaging or mailing programs into VS? I'm interested in things like Skype or Instant Messenger being embedded as tool windows. If you use (or have used) something like this, how has it affected your productivity?
I'm not sure why you want to do this? I find I already have too little space in VS.
That aside, almost every dev team I have been on now communicated via a combination of IRC, MSN, Skype etc. We have always found that a flashing toolbar is a much smaller distraction to your programming zone than a tap on the shoulder. It also means we can stick our headphones on, and faze out into focused programming land, aka "The Zone", without concern for missing co-workers trying to get your attention.
I second the fact that I would find this very annoying; I prefer to read messages on my own basis, not when someone wants to send me something, and then be forced to distract my attention from what I'm on.
That said, you could fairly trivially host some sort of messaging website (twitter perhaps, or any other) in a tab in VS. I wouldn't, but you could.