I want to do some tests on our tcl tk application regarding the user interaction. As the application has parts similar to a CAD for which every mouse movement is relevant, I would like to do something like record all events of some user interactions. My goal would be to playback these events laterwards and on every program change to discover potential changes. Or even better to assure the GUI behaves always the same and produces always the same data.
I know, that I can generate some enter motion and button events, but this would not be the same like the thousands of events generated by a real user interaction. But it is very important for me to have exactly these thousands of events.
Is there any possibility to achieve this?
It's relatively easy to record events of particular types with bind — you'll find that <ButtonPress>, <ButtonRelease>, <Enter>, <Leave>, <FocusIn>, <FocusOut>, <KeyPress> and <KeyRelease> cover pretty much everything that you are interested in — and then play them back with event generate. (You need to record quite a bit of information about each event in order to regenerate it correctly, but the underlying model is that of X events with similar names.) Assuming you're not wanting to support inter-application cut-and-paste or drag-and-drop for the purposes of recording; those complicate things a lot. You'll likely have a lot of events; recording to an SQLite database might make a lot of sense.
However, you should think carefully about which parts of the application you want to record. Does it matter if the order of two buttons in the outer shell of the application outside the CAD-like area get swapped in order? For most users, provided you're clear about what the buttons do (through clear labels and icons) it isn't very important, but for replaying recorded events it can matter hugely. Instead, for the parts of the application that are simple buttons and edit fields, I'd not record the details of them but would instead just record when the buttons are clicked and the changes to the text content of entries and so on. In effect, it's capturing higher-level events, and that's much easier to replay correctly. It's only when the user is in that main CAD area that you need the full detail.
Also, beware of changes to font sizes and screen sizes/scaling. They can change how things are laid out and may happen because of system-level alterations outside the scope of your application.
We started out the way you describe: record all those thousands of motion events, etc. Including exact timings which are extremely important for a GUI application as well.
It quickly became appearent that those recordings became too hard to maintain. They are also overly brittle in light of UI changes. Another problem where the hardcoded time values. A switch to a more powerful machine (or a cpu under load) would break the execution.
The two biggest improvements we introduced
Event compression: recognize the high-level action the user wanted to perform (like selecting a menu item). The recorded activateItem command would then perform the necessary work (event emulation) on replay.
Synchronization functions: instead of relying on a particular timing commands like waitForObject wait for an object to come into existance and become ready for interaction.
It took several years for this to work fluently, however. Including a central Object Map repository, property and screenshot verifications, high-level test descriptions in BDD and others. Feel free to take a look a the Squish for Tk product that came out of this work.
Related
If one is attempting to build a desktop program with a semi-complex GUI, especially one in which users can open multiple instances of identical GUI components such as having a "project" GUI and permitting users to open multiple projects concurrently within the main window, is it good practice to push the event listeners further up the widget hierarchy and use the event detail to determine upon which widget the event took place, as opposed to placing event listeners on each individual widget?
For example, in doing something similar in a web browser, there were no event listeners on any individual project GUI elements. The listeners were on the parent container that held the multiple instances of each project GUI. A project had multiple tabs within its GUI, but only one tab was visible within a project at a time and only one project was visible at any one time; so, it was fairly easy to use classes on the HTML elements and then the e.matches() method on the event.target to act upon the currently visible tab within the currently visible project in a manner that was independent of which project it was that was visible. Without any real performance testing, it was my unqualified impression as an amateur that having as few event listeners as possible was more efficient and I got most of that by reading information that wasn't very exact.
I read recently in John Ousterhout's book that Tk applications can have hundreds of event handlers and wondered whether or not attempting to limit the number of them as described above really makes any difference in Tcl/Tk.
My purpose in asking this question is solely to understand events better in order to start off the coding of my Tcl/Tk program correctly and not have to re-code a bunch of poorly structured event listeners. I'm not attempting to dispute anything written in the mentioned book and don't know enough to do so if I wanted to.
Thank you for any guidance you may be able to provide.
Having hundreds of event handlers is usually just a mark that there's a lot of different events possibly getting sent around. Since you usually (but not always) try to specialize the binding to be as specific as possible, the actual event handler is usually really small, but might call a procedure to do the work. That tends to work out well in practice. Myself, my rule of thumb is that if it is not a simple call then I'll put in a helper procedure; it's easier to debug them that way. (The main exception to my rule is if I want to generate a break.)
There are four levels you can usually bind on (plus more widget-specific ones for canvas and text):
The individual widget. This is the one that you'll use most.
The widget class. This is mostly used by Tk; you'll usually not want to change it because it may alter the behaviour of code that you just use. (For example, don't alter the behaviour of buttons!)
The toplevel containing the widget. This is ideal for hotkeys. (Be very careful though; some bindings at this level can be trouble. <Destroy> is the one that usually bites.) Toplevel widgets themselves don't have this, because of rule 1.
all, which does what it says, and which you almost never need.
You can define others with bindtags… but it's usually not a great plan as it is a lot of work.
The other thing to bear in mind is that Tk supports virtual events, <<FooBarHappened>>. They have all sorts of uses, but the main one in a complex application (that you should take note of) is for defining higher-level events that are triggered by a sequence of low-level events occasionally, and yet which other widgets than the originator may wish to take note of.
We update content control for every character typed in the task pane’s input field. So that user can see the live updates on the word document.
Recently we added functionality for locking content controls. And it happens as below:
User input (types a character) in a input field
We search a content control for that input field (involves context.sync)
Unlock the content control (involves context.sync)
Update value in content control (involves context.sync)
Lock back the content control (involves context.sync)
All this works nice in Word for windows without problems.
But is extremely (visibly) slow with Word for Mac (apple machines)
How should I overcome the delays happening on Mac?
As Juan mentioned in the comment, there are some important details that the team would need to investigate. Sample code would be good too.
That being said, just looking at what you describe, I think you can dramatically cut down on the context.sync() statements. Unlocking the content control, updating its value, and locking it should all be possible to do in one sync.
I have a bunch of details about minimizing sync-s in my book, "Building Office Add-ins using Office.js. Quoting one of the sections from it:
As an add-in author, your job is to minimize the number of context.sync()
calls. Each sync is an extra round-trip to the host application; and when
that application is Office Online, the cost of each of those round-trip adds up
quickly.
If you set out to write your add-in with this in principle in mind, you will
find that you need a surprisingly small number of sync calls. In fact, when
writing this chapter, I found that I really needed to rack my brain to come up with a scenario that did need more than two sync calls. The trick for
minimizing sync calls is to arrange the application logic in such a way that
you're initially scraping the document for whatever information you need
(and queuing it all up for loading), and then following up with a bunch
of operations that modify the document (based on the previously-loaded
data). You've seen several examples of this already: one in the intro chapter,
when describing why Office.js is async; and more recently in the "canonical
sample" section at the beginning of this chapter. For the latter, note that the
scenario itself was reasonably complex: reading document data, processing
it to determine which city has experienced the highest growth, and then
creating a formatted table and chart out of that data. However, given the
"time-travel" superpowers of proxy objects, you can still accomplish this task
as one group of read operations, followed by a group of write operations.
Still, there are some scenarios where multiple loads may be required. And in
fact, there may be legitimate scenarios where even doing an extra sync is the
right thing to do – if it saves on loading a bunch of unneeded data. You will
see an example of this later in the chapter.
I am trying to learn how to use BDD for our development process and I sometimes end-up writing things that implies a UI design, so for brand new development or new features, the UI does not always exists.
For example, if I say this in a scenario "When a column header is clicked" it implies that this feature is based on some sort of table or grid, but at this point we are still just writing user-stories so there is no UI yet.
That gets me confused to know at what point in the process do we come up with a UI design ?
Keep in mind, I only have read articles about BDD and I think it would help our team a lot but still very new at this! Thx!
If you write your scenarios with a focus on the capabilities of the system, you'll be able to refactor the underlying steps within those scenarios more easily. It keeps them flexible. So I'd ask - what does clicking the column get for you? Are you selecting something? What are you going to do with the selection? Are you searching for something and sorting by a value?
I like to see scenarios which say things like:
When I look for the entry
When I go to the diary for January
When I look at the newest entries
When I look at the same T-shirt in black
These could all involve clicking on a column header, but the implementation detail doesn't matter. It's the capability of the system.
Beneath these high-level scenarios and steps I like to create a screen or page with the smaller steps like clicking buttons in it. This makes it easy to refactor.
I wrote this in a DSL rather than English, but it works with the same idea - you can't tell from the steps whether it's a GUI or a web page, and some of the steps involve multiple UI actions:
http://code.google.com/p/wipflash/source/browse/Example.PetShop.Scenarios/PetRegistrationAndPurchase.cs
Hope you find it interesting and maybe it helps. Good luck!
I guess you can write around that by saying "when I sort the information by X, then..." But then you would have to adjust your scenario to remove any mention of the data being displayed in a grid format, which could lead to some rather obtuse writing.
I think it's a good idea to start with UI design as soon as you possibly can. In the case you mentioned above, I think it would be perfectly valid to augment the user story with sketch of the relevant UI as you would imagine it, and then refine it as you go along. A pencil sketch on a piece of paper should be fine. Or you could use a tablet and SketchBook Pro if you want something all digital.
My point is that I don't see a real reason for the UI design to be left out of user stories. You probably already know that you're going to build a Windows, WPF, or Web application. And it's safe to assume that when you want to display tabular data, you'll be using a grid. Keeping these assumptions out of the requirements obfuscates them without adding any real value.
User stories benefit from the fact, that you describe concrete interactions and once you know concrete data and behaviour of the system for it, you might as well add more information about the way you interact. This allows you to use some tools like Cucumber, which with Selenium enables you to translate a story to a test. You might go even further and e.g. for web apps capture all pages you start concrete story at and collect all interactions with that page resulting in some sort of information architecture you might use for documentation or prototyping and later UI testing.
On the other hand, this makes your stories somewhat brittle when it comes to UI changes. I think the agile way of thinking about this is same as when it comes to design changes - do not design for the future, do the simplest possible thing, in the future you might need to change it anyway.
If you stripped your user stories of all concrete things (even inputs) you will end up with use cases(at least in their simplest format, depends on how you write your stories). Use cases are in this respect not brittle at all, they specify only goals. This makes them resistant to change, but its harder to transfer information automatically using tools.
As for the process, RUP/UP derives UI from use cases, but I think agile is in its nature incremental (I will not say iterative, this would exclude agile methods like FDD and Kanban). This means, as you implement new story, you add to your UI what is necessary. This only makes adding UI specifics in stories more reasonable. The problem is, that this is not a very good way to create UI or more generally UX(user experience). This is exactly what one might call a weakpoint of agile. The Agile manifesto concentrates on functional software, but that is it. There are as far as I know no agile techniques for designing UI or UX.
I think you just need to step back a bit.
BAD: When I click the column header, the rows get sorted by the column I clicked.
GOOD: Then I sort the rows by name, or sometimes by ZIP code if the name is very common, like "Smith".
A user story / workflow is a sequence of what the user wants to achieve, not a sequence of actions how he achieves that. You are collecting the What's so you can determine the best How's for all users and use cases.
Looking at a singular aspect of your post:
if I say this in a scenario "When a column header is clicked" it implies that this feature is based on some sort of table or grid, but at this point we are still just writing user-stories so there is no UI yet.
If this came from a user, not from you, it would show a hidden expectation that there actually is a table or grid with column headers. Even coming from you it's not entirely without value, as you might be a user, too. It might be short-sighted, thinking of a grid just because it comes from an SQL query, or it might be spot-on because it's the presentation you expect the data in. A creative UI isnÄt a bad thing as such, but ignoring user expectations is.
im looking for suggestion in which GUI tool is most appropriate for me to use in implementing my study. im using java language. i would like the graphics to simulate a house in which graphical changes apply without user inputs from mouse or keyboards. my user input is in the form of sms. thanks in advance guys. im hoping to animate it or simulate a smart home through the conditions i had set in my program. thnaks!
Your questions is very underspecified. I will assume that you are at the early stages of producing a hand-rolled home automation programs, you probably need:
an environment to let you test the core logic of the system (i.e. "If the system is in state X and I issue command Y, what does it actually do, and will I lose the contents of my freezer?")
an environment to let you test the SMS communications module
you may need a demo mode to show prospecitve customers what it does (this is my best guess at what is being requested here)
Now (3) could fill in for (1), but is a lot more programming effort, so from the start you probably want a simple text interface to do (1).
In general, you almost certainly want a modular system: a core logic system supported by at least two input models (SMS and keyboard), three output models (text debug, graphical demo, and control-line/wireless signals for the actual hardware), and various ancillary stuff (configuration reading, saved state handling). Come to think of it, since you probably need a way to probe the current state of the system, you should make the saved state and condition probe code share a single framework as well.
I'm looking for advanced strategies for dealing with User Object Handle limits when building heavy-weight windows interfaces. Please explain how you overcame or bypassed this issue using SWT or direct Windows GUI APIs. The only thing I am not interested in is strategies to optimize widget usage as I have done this extensively and it does not solve the problem, only makes it less likely.
My Situation:
I have an SWT based GUI that allows for multiple sessions within the same parent shell and within each session their are 3 separate places where a list of user generated comments are displayed. As a user opens multiple sessions and pulls data that populates those lists, the number of user object handles can increase dramatically depending on the number of comments.
My current solutions:
1. I page the comments by default thereby limiting the number of comment rows in each session, but due to management demands, i also have what is effectively a "View All" button which bypasses this completely.
2. I custom draw all non-editable information in each row. This means each row utilizes only 2 object handles.
3. I created JNI calls which query the OS for the current usage and the Max usage. With this i can give indications to users that a crash is imminent. Needless to say, they ignore this warning.
First off, are you sure the problem isn't desktop heap vs. handle count? Each handle can consume a certain amount of Windows desktop heap. One USER handle may eat a lot of space, some very little. I'm suggesting this to make sure you're not chasing user handle counts when it's really something else. (google for Microsoft's dheapmon tool, it may help)
I've read that you can alter the maxes on handles by changing keys in the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\
CurrentVersion\Windows\ USERProcessHandleQuota and GDIProcessHandleQuota
This could be a short term fix for users.
I'd approach this by first figuring out what 2 user handles need to be maintained for each item (like 2 for each item in a listbox?). This seems suspect. User handles are for only a few top-level Windows UI objects (Windows, menus, cursors, Window positions, icons, etc...). I don't see why your widget needs to keep 2 objects around for each item (is it an icon handle??).
If you're looking to rip the whole thing apart - this sounds like a job for a virtual-mode List-View (LVS_OWNERDATA).
You should think about using windowless controls. They are designed for precisely this situation. See "Windowless controls are not magic", by Raymond Chen
Not only top-level windows, but most native controls use one user object each. See Give Me a Handle, and I'll Show You an Object for an in-depth explanation of user- and other handle types. This also means that SWT uses at least one user handle per widget, even for a Composite.
If you truly are hitting the limit of 10000 user objects per process, and you don't have a leak, then your only option is to reduce the number of widget instances in your application. I wrote a blog article about how we did this for our application.