I am currently trying to automate a Windows Forms application by using the Microsoft UI Automation Library and C#, but I have big problems concerning the performance. The Identification of single elements by using a PropertyCondition or iterating over all elements of a window takes very long (up to 4 minutes). As soon as I have a AutomationElement, everything is fine (e.g. GetCurrentPropertyValue reacts within 100ms).
The poor performance only applies to one application. I don't have access to the source but if something needs to be changed or checked, I can talk to the responsible programmer. As far as I know, some events (e.g. paint) were overwritten for the application. A typical window of the application contains about ~100 elements which are found by the FindAll method.
I also tried the COM interface of the UI automation library, which is about two times faster but this does not really solve the problem.
Does anyone have an idea how to solve this problem or experienced similar behavior?
We found the answer when we took a closer look at the main loop. In most cases Application.Run is used to start up the main window and run the application but for some reason the following code was used:
[...]
MainForm.Show();
while DoStop == false
{
System.Threading.Thread.Sleep(10);
Application.DoEvents();
}
[...]
As the Microsoft UI Automation Library uses window messages, all the System.Threading.Thread.Sleep(10); summed up and made the object detection become really slow. This does not happen, if Application.Run is used.
Related
Was wondering if this: https://code.visualstudio.com/docs/editor/codebasics could be implemented with WinApi or by DLL calls/injection globally in every application?
Which api call could be relevant to get me started?
This cannot be done. There are too many problems that need to be solved for which there is no general solution.
The standard carets have a hard limit of one caret per queue. With that in mind you would now have to solve not one, but two problems: Getting a custom caret implementation, and fighting the system-provided one.
That might still sound doable, but even just getting your custom-rendered carets injected into foreign windows isn't possible. There is no infrastructure in the system that allows you to safely tap into the rendering of arbitrary (or even standard) controls you do not own.
Now even if you got a solution to all of the above, how would you communicate those multiple selection marks back to client code? EM_GETSEL is strictly limited to one selection mark at most.
So far, this was mostly just about standard controls. Things won't get any easier with custom control implementations. WPF, to my knowledge, uses a pretty much closed down control library, that doesn't even provide the customization points of Windows' common controls. Same goes for UI toolkits like Qt. While open source, Qt doesn't allow for any external customization.
I'm sure there are more problems that don't have a general solution. While not an exhaustive list, the above problems prevent implementation of multi-selection in arbitrary UIs outside your control.
Can someone explain me why TWebBrowser control is working so slow on all XE editions of Delphi including XE5 and possibly XE6? To test this you need to create a new Delphi project and put TWebBrowser control in it. On form show event, navigate to this website:
http://ie.microsoft.com/testdrive/Performance/setImmediateSorting/Default.html
Please test this on Windows 7 or later. When navigation is complete, run setImmediate test and watch the results. It will take huge amount of time to complete the test. It will take about a minute to finish this.
When you open true Internet Explorer browser and do the same thing - test will be completed instantly (~200 miliseconds).
Some additional wierd informations:
When you recreate this procedure on old versions of Delphi (Delphi 7 to be precise) the web-control works as fast as it should be working and test is completed instantly. But the HTML5 speed test will still works slow (alternative test on this page).
Another weird thing is, the same slow behavior can be seen on C++ Builder but not in Visual Studio products. Is Microsoft deliberately slowing down the TWebBorwser in Embarcadero products?? I can't belive this.
I was trying to overcome this problem with diffrent methods such as:
Trying different feature options in registry such as:
FEATURE_GPU_RENDERING,
FEATURE_BROWSER_EMULATION (11001),
FEATURE_ALIGNED_TIMERS (undocumented option),
FEATURE_ALLOW_HIGHFREQ_TIMER (undocumented option),
Setting timerBeginPeriod(1) - no effect.
Please, if someone have any clue how to fix this issue - share this information with me.
UPDATE1
I made standalone test app if anyone cares. It can be downloaded here: http://mp.org.pl/download/ietest.zip It contains source and exe app with htm file. HTM file contains some js procedure that works 10 times faster in standalone IE than in TWebBrowser control. It uses setImmediate as a test (the same procedure used in test described above). But it can be easier for testing this way.
I can also see the behavior described (in your original post and in the comments). I have a few thoughts, but not necessarily an answer.
One should expect some difference in performance between the WebBrowser control and IE, in part because your Delphi app will need to build in support for certain features/APIs that IE supports out of the box.
For example, the WebBrowser control fires notifications related to tabbed browsing (old, but relevant), but it does not intrinsically handle those notifications or update the UI. You have to respond to the notifications and draw the tabs yourself. By default, IE is hardware accelerated and uses certain Windows APIs that may not be directly supported by Delphi's VCL (for resource/performance) reasons. (Hardware acceleration could account for some of the performance differences you've noticed.)
(And, for the record, I don't believe a list of differences between IE and the WebBrowser control was ever documented. I certainly don't remember seeing one in the portfolio.)
Also, the default values for various feature controls vary between IE and applications hosting the WebBrowser control. Part of the reason for this stems from the idea that IE needs to highlight performance over compatibility whereas applications generally need to emphasize compatibility over performance. You may wish to review the feature control reference to see if there are other FCKs you need to enable for your app.
Second, your loops are very tight, perhaps too tight. You've got one request piling on earlier requests and you're not really leaving much room for processing, even with setImmediate. (IIRC, we're not really supposed to use anything smaller than 250ms for setInterval without risking performance hits from the sheer number of requests.) The remarks in the setImmdiate ref. page provide some guidance, as does this article on requestAnimationFrame.
One reason why dragging the window appears to improve performance may due to the priority of window drag repaint requests. They may be forcing your loops to hold long enough (or even break) to allow other events to process. Hard to say without with tracing the system with a debugger.
Have you ever had to add application.processMessages() to your Delphi apps in order to allow the system a chance to handle the work you've already assigned? A similar need may be coming into play given the nature of your test.
Performance testing and timing is a tricky thing. You need to make sure the test isn't imposing so much overhead that it interferes with the actual work you're trying to perform.
Finally, there were some questions about the document mode of the page as it's loaded into your project. When I first started messing around with your sample, I couldn't get project4 to load slowtest.html in anything other than IE5 quirks mode (notoriously slow). Here's what eventually started working for me:
<!DOCTYPE html>
<!-- saved from url=(0023)http://www.contoso.com/ -->
<html>
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<script type="text/javascript">
...
(Note, I deleted your initial doctype declaration and rewrite it to resolve a syntax error that was being reported by the F12 tools debugger.)
A few style key points here:
I used a mark of the web to load the page in the Internet zone. I find this makes it easier to load the page in edge mode, as pages in the intranet zone are loaded in compatibility view by default (unless you map the zones differently).
The x-ua-compatible header needs to be one of the first in the head block. It can follow title, but not much else.
Stylistically, elements need to be specified in lower case these days. There's a possibility that not following the current conventions forces the parser to fall back to an earlier rendering that supports the conventions.
Once I was able to control the documentMode at runtime, I found results I expected: older document modes ran more slowly. I also found that using requestAnimationFrame instead of setImmediate led to even better performance, but also surfaced the timing issue almost immediately.
In the end, this may be a case where the test highlights a problem, but not necessarily one you're trying to solve. (Insert Inigo meme here.) I get that you're trying to resolve a bottleneck. Are you sure you've found the correct bottleneck?
You may not be able to replicate the same performance of the native browser, but perhaps you can refactor the code to perform adequately without the extra overhead? Is there anything that might be better handled with a worker or some other implementation technique?
Hope this helps...
For about a year a half, I've been working with SilkTest, which is a GUI automation tool, for both desktop and web applications. It simulates mouse and keyboard inputs, which eventually simulate end user behaviour. However, I find that it is a bit flaky; Button.Click() or DialogBox.Close() method calls that work just fine 9 times in a row seem to fail on a 10th call, only to go back working on the 11th. Normally I would just chalk this up to a quirk with SilkTest (or the application under Test, or the OS, or what have you) but then I see that there are similar issues with other GUI automation tools like Selenium:
Selenium Click() fails with Anchor Elements
Selenium Click() fails clicking button object
I know that for desktop apps, each GUI control/dialog has a tag element associated with it (at least in Windows-based GUIs) and that for web pages there is the domain object model hierarchy of page elements. My guess is that these tools sometimes run into issues navigating these hierarchies and finding unique elements and controls. But what is going on here? SilkTest is a relatively old, commercial software package while selenium is relatively new, open source and constantly evolving. The fact that they both can have similar problems raises a couple of flags with me.
Also, is this the case with other GUI test tools? Or have I just had a somewhat unusual experience?
There are 2 things here that you are talking about, first the concept of finding an object in the application under test that you want to automate. Your description of how SilkTest (and other tools) does this is quite accurate, i.e. as long as there is something that the automation software can use to identify the control then you are fine.
The second thing is why does the automation itself fails randomly, since the tool has not reported that it could not find the control then it must think that it sent the appropriate action to the application, e.g. a Click or a Type. This could be that the application is not ready to accept the action that you are sending it, this is similar to you attempting to click on something "before it was ready", in this case the application can decide to buffer the input or to discard the input.
So, how do you fix this? One way would be to use the capabilities of the tool to try to work out when the application is ready for input rather than sending it a stream of input blindly. SilkTest has capabilities that allow for you to do this (as does TestPartner). I cannot comment on Selenium as it is something I have not used.
A simple way of testing this would be to insert a pause for a couple of seconds before the offending action, then run this in a loop to see whether this solves the problem, if this is the case then it is your problem. If this does not fix the issue then there is something else going on that you need to contact the vendor of the testing tool.
Remember that applications are getting more and more complex, i.e. multi-threading, communications, any one of these could cause the automatic syncronisation to fail causing actions to fail.
Hope that helps.
My application has several large forms with lots of images which dramatically increases the size of the built executable. Over time, it seems that the startup performance becomes sluggish and it doesn't seem to be getting any better.
If I put all of the forms besides the main form in a separate dll, would it alleviate some of the pressure put on the application during startup?
I'd test it myself, but I have A LOT of forms and I don't want to do it unless someone can confirm that such an action will prove to be useful.
Many factors can affect startup performance. Have you used any tools to prove that it's the images?
For a start, go through these tips:
http://devcomponents.com/blog/?p=361
And consider using multithreading to load bigger objects in the background.
I'm not quite sure about that, but if I were you I would use the Profiler when it comes to improving performance.
Before I go guessing what's wrong, I consult with it and work my way up, because it tells me which methods and classes are costing the most in my code.
Another tip that may be useful: This reduced my application's startup time from 2 minutes to <10 seconds on a low-end thin client. Use NGEN to generate a precompiled native image of your assemblies.
I'm wondering if you were to use MEF and Lazy load, then when you actually need the module (Form) instantiate by calling .Value.
There are a couple of things I do with applications containing a lot of forms:
Create a UI .exe: basically only my forms
Create a backend .dll: everything that does the work behind the UI.
Are the images actually included in the .dll? If so, I would actually put my images into a .dll separate from the UI.
Given that the images are for toolbars, I wouldn't split them out as resources. I'll still stand fast on my advice to split into multiple .dlls.
As others said, profile, don't guess.
Not just any profiler will do.
Here's a user (besides me) who discovered random pausing on his own.
You say the "intense" methods are all in dlls you don't have source code for - that's typical and normal.
What you need to know is which statements in your code are requesting the time to be spent, and they can't be restricted to CPU-only time.
Most profilers don't tell you this, but random-pausing does.
If you're interested, here's a recent discussion of the issues.
How to implement multi-threading in visual basic 6.0. It would be great if someone can give an example.
VB6 is not a really good environment for multi-threaded applications. There is no out-of-the-box support, you need to delve into standard WinAPI functions. Take a look at this article, which provides quite a comprehensive sample:
http://www.freevbcode.com/ShowCode.Asp?ID=1287
On several projects I have implemented asynchronous processing in VB6 using multiple processes. Basically having a worker thread within an active exe project that is separate from the main process. The worker exe can then be passed whatever data it needs and started, raising back an event to say it's finished or there is data for the main process.
It's a more resource hungry (an extra process rather than a thread) but VB6 is running in a single threaded apartment and doesn't have any built in support for starting new threads.
If you really need to have multiple threads within one process I'd suggest looking at using .net or VC6 rather than VB6.
If the problem that you are trying to solve is a long calculation and you want to keep the UI responsive, then one possibility is to frequently call the DoEvents function within your long calculation. This way, your program can process any Windows messages, and thus the UI will respond to user commands. You can also set up a Cancel button to signal your process that it needs to end.
If you do this, then you will need to be careful to disable any controls that could cause a problem, such as running the long process a second time after it has started.
Create "Active X" controls to manage your code. Each control has its own thread. You can stack multiple controls doing the same thing, or have individual controls doing unique things.
EG, You make one to download a file from the net. Add ten controls and you have ten individual threaded downloads running, independent of the thread which the actual program is running. Essentially, they are all just interactive, windows, controlled by an instanced mini-dll program.
Can't get any easier than that. You can throttle them, turn them on and off, as well as create more, or remove them, as needed. (Indexing just like any other of the "Objects", on a form. Which are all just active-x controls, which are simply managed by the vb-runtime dlls.)
You can use the Interop Forms Toolkit 2.0 for multithreading in VB6. The Toolkit
allows you to take advantage of .NET features without being forced onto an upgrade pat. Thus you can also use .NET User Controls as ActiveX controls in VB6.