I'm struggling with AutoIt regression tests. AutoIt was chosen because requirement is 100% blackbox GUI testing (no intervention to tested project). This solution has nevertheless got some issues, with which I need help.
Changing tabs in application - Because GUI class isn't SysTabControl32, but SWT_Window, example code from GUITab.au3 UDF doesn't work. Current solution is focusing on component and then navigating through tabs with arrow keys. This has a bad influence on test performance (and I don't even want to imagine a future possibility of some tab being disabled).
Timeouts - When tabs are changing, tests have to delay before they can proceed. The shorter the delay, the bigger the probability of test failure (app wasn't ready for test to continue). This leads to big delays before actions.
Instance numbers - Identifying instances of controls is a problem. When I wrote some test button OK it had instance number 9. When some buttons were added to the form I had to rewrite the test, because instance number of OK button changed because of this.
These three are most important.
Changing technology of testing would be hard because of big amount of already written tests. But I would like to write new test in a better way. Sikuli has problems getting text from screen and SWTBot has dependencies in tested projects.
Our tests run for 20 hours and when GUI layout changes I need to edit almost every test (instance number problem). Can anybody suggest a solution or workaround for ultra reliable blackbox testing?
Related
For about a year a half, I've been working with SilkTest, which is a GUI automation tool, for both desktop and web applications. It simulates mouse and keyboard inputs, which eventually simulate end user behaviour. However, I find that it is a bit flaky; Button.Click() or DialogBox.Close() method calls that work just fine 9 times in a row seem to fail on a 10th call, only to go back working on the 11th. Normally I would just chalk this up to a quirk with SilkTest (or the application under Test, or the OS, or what have you) but then I see that there are similar issues with other GUI automation tools like Selenium:
Selenium Click() fails with Anchor Elements
Selenium Click() fails clicking button object
I know that for desktop apps, each GUI control/dialog has a tag element associated with it (at least in Windows-based GUIs) and that for web pages there is the domain object model hierarchy of page elements. My guess is that these tools sometimes run into issues navigating these hierarchies and finding unique elements and controls. But what is going on here? SilkTest is a relatively old, commercial software package while selenium is relatively new, open source and constantly evolving. The fact that they both can have similar problems raises a couple of flags with me.
Also, is this the case with other GUI test tools? Or have I just had a somewhat unusual experience?
There are 2 things here that you are talking about, first the concept of finding an object in the application under test that you want to automate. Your description of how SilkTest (and other tools) does this is quite accurate, i.e. as long as there is something that the automation software can use to identify the control then you are fine.
The second thing is why does the automation itself fails randomly, since the tool has not reported that it could not find the control then it must think that it sent the appropriate action to the application, e.g. a Click or a Type. This could be that the application is not ready to accept the action that you are sending it, this is similar to you attempting to click on something "before it was ready", in this case the application can decide to buffer the input or to discard the input.
So, how do you fix this? One way would be to use the capabilities of the tool to try to work out when the application is ready for input rather than sending it a stream of input blindly. SilkTest has capabilities that allow for you to do this (as does TestPartner). I cannot comment on Selenium as it is something I have not used.
A simple way of testing this would be to insert a pause for a couple of seconds before the offending action, then run this in a loop to see whether this solves the problem, if this is the case then it is your problem. If this does not fix the issue then there is something else going on that you need to contact the vendor of the testing tool.
Remember that applications are getting more and more complex, i.e. multi-threading, communications, any one of these could cause the automatic syncronisation to fail causing actions to fail.
Hope that helps.
I am currently trying to automate a Windows Forms application by using the Microsoft UI Automation Library and C#, but I have big problems concerning the performance. The Identification of single elements by using a PropertyCondition or iterating over all elements of a window takes very long (up to 4 minutes). As soon as I have a AutomationElement, everything is fine (e.g. GetCurrentPropertyValue reacts within 100ms).
The poor performance only applies to one application. I don't have access to the source but if something needs to be changed or checked, I can talk to the responsible programmer. As far as I know, some events (e.g. paint) were overwritten for the application. A typical window of the application contains about ~100 elements which are found by the FindAll method.
I also tried the COM interface of the UI automation library, which is about two times faster but this does not really solve the problem.
Does anyone have an idea how to solve this problem or experienced similar behavior?
We found the answer when we took a closer look at the main loop. In most cases Application.Run is used to start up the main window and run the application but for some reason the following code was used:
[...]
MainForm.Show();
while DoStop == false
{
System.Threading.Thread.Sleep(10);
Application.DoEvents();
}
[...]
As the Microsoft UI Automation Library uses window messages, all the System.Threading.Thread.Sleep(10); summed up and made the object detection become really slow. This does not happen, if Application.Run is used.
I know this question looks a lot like this one, but I don't have enough rep points to comment seeking further clarification VS2010 Coded UI Tests vs. Web Performance test (Whats the difference??)
Tom E. gives a great explanation, but I'm still a bit vague on one point. I see why Coded UI tests cannot be replaced by Web Performance tests (the extra resources needed for a browser interface) but why can Web Performance tests not replace Coded UI tests?
If you record a webperf test and generate the code from that, couldn't you add validation and extraction rules (inspecting the DOM) to achieve the same result as a Coded UI test without the overhead of the browser?
I realize that this wouldn't be exactly the same as testing in different browsers, but is there a reason this wouldn't at least test whether you're receiving the appropriate response from the server?
Thanks!
Dave, good point. I think you would see the difference fairly quickly if you were trying to build an automated functional test suite (think 500 tests or more) with VS web performance tests and having to parse the DOM for querying and interacting with the application. You would essentially be writing your own Coded UI test playback mechanism. You could do it without the Coded UI test functionality, but it would be quite painful. The level of pain would be dependent on how many test cases you need to automate and how many screens there are in your app and how complex the interactions are.
I am just jumping into testing of a GUI-heavy app written in C++(MFC) and C# (WinForms, WPF).
I have played with coded ui tests briefly, and I am happy with what I see. The trouble is that my program is non-trivial, so this does not work for me for 2 reasons:
VS 2010 Coded UI Test - Launch Referenced Application
There are 3 programs to launch in certain order, and I need to log in to one of them. Timing is important; I need to make sure that they have come up.
Start-up time is long, so I want to make sure that this set up is only done once per project.
When the project is done, I want to cleanly destroy this thing.
For each test I would want to make sure that the windows is active and is in the forefront at the beginning of each test.
What are some good approaches? If you find that this question is too vague, please let me know what it is missing.
There is code that gets generated for the coded UI tests. You can edit the coded UI test and add your own code to wait for things to occur before letting the next recorded step to take place. If you want to bring a certain application to the forefront, you can use Process.GetProcessesByName to get process information about your running application(s) and get its mainwindow handle to bring it to the front.
I you want to delay your test and wait some tasks (for example log-in or server response) you can check this article.
http://msdn.microsoft.com/en-us/library/gg316453.aspx
I would like to know how to go about doing regression testing effectively for a web application.
I m spending a lot of time in regression testing for every build.
I also hear that automated GUI testing is bad and not worth it.
Is there any better way of automating regression testing. if so any good tools to automate GUI tests.
My web app is designed for stupid IE. cant help it, but that how it is.Any good tool that can help me?
I know that we must keep GUI layer thin and all, but I m not a GUI expert or anything , but i need to test it properly and effectively.
Thanks all
Try looking into Cucumber or Selenium
There is also a screencast on it over at railscasts.
That should really help.
You can use standard unit tests and integration tests for the non-GUI components, of course. For the GUI, it's worth looking at UI automation packages.
Many of them are pretty terrible, but a lot of the horror stories come from people using them incorrectly. If you can stay away from specifying click locations by (x,y) coordinates, and you can detect when state transitions happen, rather than using delay(), GUI testing can be effectively automated.
As long as your webapp works with Firefox, too, I recommend using Selenium for that scenario. It allows you to record your test cases inside the browser (by means of a Firefox plugin) but has Internet Explorer support for playback.
If Firefox for recording tests is not an option, you could have a look at the WebDriver project. It uses a code based approach to test specification an works well with TDD, too.
You might also want to have a look at CubicTest, which is a Eclipse plugin for graphically defining Selenium and Watir test cases.