My goal is to screen-scrape a portion of a program which constantly updates with new text. I have tried OCR with Tesseract but I believe it would be much more efficient to somehow intercept the text if possible. I have attempted using the GetWindowText() function, but it only returns the window title. Using Window Detective I have determined that whenever the window updates in the way I wish to capture, a WM_PAINT message is reliably sent to the window.
I have looked a bit into Windows API Hooks, but it seems that most of these techniques involving DLL injection are intended at sending new messages, not accessing the content of already sent messages.
How should I approach this problem?
When you say 'screen-scrape', is that what you really mean? Reading your post, it sounds like you actually want to get at the text in the child window or control in question - as text, and not just as a bitmap. To do that, you will need to:
Determine which child window or control actually contains the text you want to get at. It sounds like you may have already done that but if not, the tool of choice is generally Spy++. (Please note: the version of Spy that you use must match the 'bitness' of your application.)
Then, firstly, try to figure out whether the text in that window can be retrieved somehow. If it's a standard Windows control (specifically EDIT or RICHEDIT) then there are documented ways to do that, see MSDN.
If that doesn't pan out, you might have some success hooking calls to ExtTextOut(), although that's not a pleasant proposition and I think you might struggle to achieve it. That said, I believe the accepted way (in some sense of the word 'accepted') is here.
With reference to point 3, even if you achieve it, how would you know whether any particular call to ExtTextOut() was drawing to the window you're interested in? Answer, most likely, HWND WindowFromDC().
I hope that helps a little. Please don't come back at me with a bunch of detailed questions about how this might apply to your particular use-case. I'm not really interested in that, these are just intended as a few signposts.
I'm trying to implement a syntax-coloring text editor that also does things like insert whitespace at the start of a new line for you, or replace text with text attachments.
After perusing the docs again after a previous implementation had issues with undoing, it seems like the recommended bottleneck for this is NSTextStorageDelegate's textStorage(_,willProcessEditing:,range:,changeInLength:) method (which states that Delegates can change the characters or attributes., whereas didProcessEditing says I can only change attributes). This works fine, except that whenever I actually change attributes or text, the text insertion mark moves to the end of whatever range of text I modify (so if I change the style of the entire line, the cursor goes at the end of the line).
Does anybody know what additional call(s) I am missing that tell NSTextStorage/NSTextView not to screw up the insertion mark? Also, once I insert text, I might have to tell it to move the insertion mark to account for text I've inserted.
Note: I've seen Modifying NSTextStorage causes insertion point to move to the end of the line, but that assumes I'm subclassing NSTextStorage, so I can't use the solution there (and would rather not subclass NSTextStorage, as it's a semi-abstract subclass and I'd lose certain behaviours of Apple's class if I subclassed it).
I found out the source of the problem.
And the only solution that will work robustly based on reasons inherent to the Cocoa framework instead of mere work-arounds. (Note there's probably at least one other, metastable approach based on a ton of quick-fixes that produces a similar result, but as metastable alternatives go, that'll be very fragile and require a ton of effort to maintain.)
TL;DR Problem: NSTextStorage collects edited calls and combines the ranges, starting with the user-edited change (e.g. the insertion), then adding all ranges from addAttributes(_:range:) calls during highlighting.
TL;DR Solution: Perform highlighting from textDidChange(_:) exclusively.
Details
This only applies to a single processEditing() run, both in NSTextStorage subclasses and in NSTextStorageDelegate callbacks.
The only safe way to perform highlighting I found is to hook into NSText.didChangeNotification or implement NSTextDelegate.textDidChange(_:).
As per #Willeke's comments to the OP's question, this is the best place to perform changes after the layout pass. But as opposed to the comment thread, setting back NSText.selectedRange does not suffice. You won't notice the problem of post-fixing the selection after the caret has moved away until
you highlight whole blocks of text,
spanning multiple lines, and
exceeding the visible (NSClipView) boundaries of the scroll view.
In this rare case, most keystrokes will make the scroll view jiggle or bounce around. But there's no additional quick-fix against this. I tried. Neither preventing sending the scroll commands from private API in NSLayoutManager nor avoiding scrolling by overriding all methods with "scroll" in them from a NSTextView subclass works well. You can stop scrolling to the insertion point altogether, sure, but no such luck getting a solid algorithm out that does not scroll only when you perform highlighting.
The didChangeNotification approach does work reliably in all situations I and my app's testers were able to come up with (including a crash situation as weird as scrolling the text and then, during the animation, replacing the string with something shorter -- yeah, try to figure that kind of stuff out from crash logs that report invalid glyph generation ...).
This approach works because it does 2 glyph generation passes:
One pass for the edited range, in the case of typing for every key stroke with a NSRange of length 1, sending the edited notification with both [.editedCharacters, .editedAttributes], the former being responsible for moving the caret;
another pass for whatever range is affected by syntax highlighting, sending the edited notification with [.editedAttributes] only, thus not affecting the caret's position at all.
Even more details
In case you want to know more about the source of the problem, I put more my research, different approaches, and details of the solution in a much longer blog post for reference. This here, though, is the solution itself. http://christiantietze.de/posts/2017/11/syntax-highlight-nstextstorage-insertion-point-change/
The above accepted answer with the notification center worked for me, but I had to include one more thing when editing text. (Which may be different from selection).
The editedRange of the NSTextStorage was whack after the notification center callback. So I keep track of the last known value myself by overriding the processEditing function and using that value later when I get the callback.
override func processEditing() {
// Hack.. the editedRange property when reading from the notification center callback is weird
lastEditedRange = editedRange
super.processEditing()
}
I am writing a Mac application that provides a "test" like function. This application (through a connection with a server). Basically the application will give the students a story to read, followed by a series of questions (also from the server) where the user can (attempt) to select the correct answers, and send the result back to the server to be verified.
Implementing the "story" part was easy. Just send all of the text to a NSTextView. I had been planning to implement the "select your answer" as programmatically created NSSwitchButtons. However, some of the possible answers might take up more than one line. I have not been able get (any) NSButton class to wrap text based on the frame size, and there doesn't seem to be an easy way to override NSButtonCell to allow the text to wrap.
What other Cocoa class(es) should I use to accomplish this task? I need to have a check-box interface (so that people can select one or more possible answers, and the answers can be an arbitrary length - within reason!) Ideally it would also be easy to use so that it will be easy to programatically layout the answers as well. (Some problems may only have 2 choices, while others may have 5+) I can't imagine I'm the only one who needs this type of functionality
(Oh...since a picture is worth 1,000 words, I've attached a screen-shot of my app below with some answer text running off of the screen)
An NSButton will respect explicit linefeed characters embedded in the text, but I suppose that would not meet your needs. An alternative would be to have a static text item next to a checkbox with no title. Of course, if you want to be able to toggle the checkbox by clicking the text, you would have a little more programming to do.
I have a Cocoa application managing a collection of objects. The collection is presented in an NSCollectionView, with a "new object" button nearby so users can add to the collection. Of course, I know that having a "delete object" button next to that button would be dangerous, because people might accidentally knock it when they mean to create something. I don't like having "are you sure you want to..." dialogues, so I dispensed with the "delete object". There's a menu item under Edit for removing an object, and you can hit Cmd-backspace to do the same. The app supports undoing delete actions.
Now I'm getting support emails ranging from "does it have to be so hard to delete things" to "why can't I delete objects?". That suggests I've made it a bit too hard, so what's the happy middle ground? I see applications from Apple that do it my way, or with the add/remove buttons next to each other, but I hate that latter option. Is there another good (and preferably common) convention for delete controls? I thought about an action menu but I don't think I have any other actions that would go in it, rendering the menu a bit thin.
Update I should also point out that delete should be an infrequent option - the app is in beta so users are trying out everything. This is a music practise journal, so creating new things to practise happens every so often (and is definitely needed when you start out using the app), but deleting them is not so frequent.
Drew's remark is always your first consideration. All other things being equal, I'm not a fan of making deletion as easy as creation; it's a dangerous and comparatively rarer action, and the UI should reflect that fact. However, not having an explicit delete control can indeed lead to support enquiries (the same happened in MoneyWell after the minus buttons were removed). The issue is that you won't hear from the people who avoided accidental deletion by hitting a too-close-to-the-plus deletion control; those people are happy and quiet. You will, however, hear from those who can't immediately find a button to click for deletion, even though almost all of Apple's applications have no such control.
If you feel that you need explicit UI for deletion, I think you can find a middle ground. The problem with deletion controls is accidental triggering, and the conventional "solution" to that problem is a confirmation alert. The problem with that is how intrusive and jarring they are, because they're modal. iPhone OS can teach us a lesson here: you can make confirmation entirely contextual and non-modal.
Examples are row-deletion (swipe to put the row into its "are you really sure you want to delete?" state, which visually tends to slide a red Delete button into view), then interact again (by tapping Delete) to actually confirm the action. There's a similar model on the App Store whereby tapping the price button changes it into a Purchase button; it's essentially an inline, non-modal confirmation. The benefit is that if you tap anywhere else (or perhaps wait a while), the control returns to its normal state on its own - you don't need to explicitly dismiss it before continuing work.
Perhaps that sort of approach (non-modal change as a sort of inline confirmation) can get rid of the support queries by making deletion controls explicit, but also patch up some of your reasonable concerns about intrusive confirmation.
I would say this depends on how important deletion is to the particular task. Is it something that the user has to do often, or very rarely. If it is rare, delete should just be left as an Edit menu option, and perhaps as backspace (Why cmd-backspace? If you can just have backspace, you probably won't get as many queries.)
As with everything in interface design, my take is to apply an 80-20 rule. If something belongs to the 20% of most used functionality, it should be exposed directly in the interface. If it is in the other 80%, you can hide it deeper (eg in a menu, action menu etc).
A + button is definitely in the top 20% --- you can't do anything without it --- whereas a delete is usually not a common operation, and is destructive, so can probably better be hidden away a bit.
The usual solution to this problem is to put the [+] and [-] buttons next to each other (see, for example, the Network pane in System Preferences). I generally find those buttons large enough that I don't hit the wrong one by mistake, although I can see that potentially being a problem.
If that option doesn't suit you, maybe take inspiration from Safari: put an 'x' inside the selected (or hovered) item.
Since your app supports undoing of deletion, I would suggest that you err on the side of making deleting stuff easy (at the expense of making it too easy) and make it obvious that these mistakes are easily undo-able. GMail does a decent job of that.
HTH.
How frequently is delete needed? Does the data and the user's expectation encourage deleting this data often? (is it a list of tasks, for example)? If so i'd certainly include a contextual action menu, even if Delete was the only option.
Cmd + Backspace may be a little unusual for people too - I know it's used in other places on OSX, but those places also provide context menus to expose the delete - i'd be surprised is every user knows about Cmd + Backspace, so i'd probably change it to Backspace (you do have undo support, so you're covered there).
Finally, and hopefully I don't sound like a git, but it suggests that the built-in help doesn't offer enough guidance on this - might be worth revising it?
Matt gave pretty much the same answer I was going to write.
Note that when you delete the object, you should animate it away: this provides valuable visual feedback: the animation (about 1/3 of a second is good) is long enough to catch the user’s eye, and they’ll see the object disappearing. If the object just disappeared without animating, the user would notice that something had changed instantaneously in the list, but would be less certain what it was. The animation reinforces the meaning of the delete button in the user’s mental model.
This is inspired by the question OK-Cancel or Cancel-OK?.
I remember reading somewhere about the concept of switching OK-Cancel/Cancel-OK in certain situations to prevent the user from clicking through information popups or dialog boxes without reading their content. As far as I remember, this also included moving the location of the OK button (horizontally, left to right) to prevent the user from just remembering where to click.
Does this really make sense? Is this a good way to force the user to "think/read first, then click"? Are there any other concepts applicable to this kind of situation?
I am particularly thinking of a safety-related application, where thoughtlessly pressing OK out of habit can result in a potentially dangerous situation whereas Cancel would lead to a safe state.
Please don't do this unless you are really, really, really sure it's absolutely required. This is a case of trying to fix carelessness and stupidity by technological means, and that sort of thing almost never works.
What you could do is use verbs or nouns instead of the typical Windows OK / Cancel button captions. That will give you an instant attention benefit without sacrificing predictability.
NOOOOOOOOOOOO!
In one of our products we have a user option to require Ctrl+Click for safety related commands.
But startling the user with buttons that swap place or move around is bad design in my book.
NO. If you make it harder for the user to click OK by mistake and force them to think, they will still only think harder about how to click OK -- they will not think about the actual thing they're trying to carry out. See usability expert Aza Raskin's article: Never use a warning when you mean Undo. Quote:
What about making the warning
impossible to ignore? If it’s
habituation on the human side that is
causing the problem, why not design
the interface such that we cannot form
a habit. That way we’ll always be
forced to stop and think before
answering the question, so we’ll
always choose the answer we mean.
That’ll solve the problem, right?
This type of thinking is not new: It’s
the
type-the-nth-word-of-this-sentence-to-continue approach. In the game Guild Wars, for
example, deleting a character requires
first clicking a “delete” button and
then typing the name of the character
as confirmation. Unfortunately, it
doesn’t always work. In particular:
It causes us to concentrate on the unhabitual-task at hand and not on
whether we want to be throwing away
our work. Thus, the
impossible-to-ignore warning is little
better than a normal warning: We end
up losing our work either way. This
(losing our work) is the worst
software sin possible.
It is remarkably annoying, and because it always requires our
attention, it necessarily distracts us
from our work (which is the second
worst software sin).
It is always slower and more work-intensive than a standard
warning. Thus, it commits the third
worst sin—requiring more work from us
than is necessary.
[If you want a Microsoftish one, this one by a .NET guy on MSDN says the same thing!]
If you must use a dialog, put descriptive captions on the buttons within the dialog.
For example, instead of OK and Cancel buttons, have them say "Send Invoice" and "Go Back", or whatever is appropriate in the context of your dialog.
That way, the text is right under their cursor and they have a good chance of understanding.
The Apple Human Interface Guideline site is a great reference, and very readable. This page on that site talks about Dialogs.
Here is an example image:
(source: apple.com)
No, it doesn't make sense. You're not going to "make" users read. If the decision is that crucial, then you're better off finding a way to mitigate the danger rather than handing a presumed-careless user a loaded gun.
Making the "safe" button default (triggered by enter/spacebar/etc.) is a good idea regardless, simply because if they surprise the user then a keystroke intended for the expected window won't accidentally trigger the unexpected action. But even in that scenario, you must be aware that by the time the user has realized what they've done, the choice is already gone (along with any explanatory text on the dialog). Again, you're better off finding another way to give them information.
What I've done in some instances was to compare the time of the message box being shown with the time of it being dismissed. If it was less than 'x' amount of seconds, it popped right back up. This forced them, in most cases, to actual read what was on the screen rather than just clicking through it blindly.
Fairly easy to do, as well....
Something like this:
Dim strStart As DateTime = Now
While Now < strStart.AddSeconds(5)
MessageBox.Show("Something just happened", "Pay Attention", MessageBoxButtons.OK)
If Now < strStart.AddSeconds(5) Then strStart = Now Else Exit While
End While
At the end of the day you can't force a user to do something they're unwilling to do... they will always find a way around it
Short cut keys to bypass the requirement to move the mouse to a moving button.
Scrolling down to the bottom of the EULA without reading it to enable to continue.
Starting the software and then going to get their cup of tea while waiting for the nag screen to enable the OK button.
The most reliable way I've seen this done is to give a multiple choice question based on what is written. If they don't get the answer correct, they can't continue... of course after a couple of times, they'll realise that they can just choose each of the answers in turn until the button enables and then click it. Once again meaning they don't read what was written.
You can only go so far before you have to put the responsibility on the user for their actions. Telling the user that their actions are logged will make them more careful - if they're being held accountable, they're more likely to do things right. Especially if there's a carefully crafted message that says something like:
This is being logged and you will be held accountable for any
repercussions of this decision. You have instructed me to delete
the table ALL_CORPORATE_DATA. Doing so will cause the entire company's
database to stop working, thus grinding the whole company to a halt.
You must select the checkbox to state that you accept this responsibility
before you can choose to continue...
And then a checkbox with "Yes, I accept the responsibility for my actions" and two buttons:
"YES, I WANT TO DELETE IT" this button should only be enabled if the checkbox is checked.
"OH CRAP, THAT'S NOT WHAT I MEANT AT ALL" this button can always be enabled.
If they delete the table and the company grids to a halt, they get fired. Then the backup is restored and everyone's happy as Larry [whoever Larry is] again.
Do NOT do it, please. This will have no positive effect: You are trying to AVOID people's clicking OK instead of Cancel, by making them potentially click Cancel instead of OK (okay, they may try again). But! you might as well achieve people's clicking OK when they really want to cancel and that could be a real disaster. It's just no good.
Why not reformulate the UI to make the OK the "safe choice"?
The problem is better solved with a combination of good feedback, communication of a system model and built-in tolerance.
In favor of the confirmation mechanism speaks the simplicity of implementation. From programmer's point of view it's the easiest way of shifting responsibility onto user: "Hey, I've asked you if you really want to shoot yourself into the foot, haven't I? Now there is no one to blame but yourself..."
From user point of view:
There is a productivity penalty of having to confirm operation twice every time even though actual mistakes take up just a fraction of total number of actions, any switching of buttons, breaking the habitual workflow or inserting a pause into confirmation just increases the penalty.
The mechanism doesn't really provide much safety net for frequent users whose reflexes work ahead of the concious mind. Personally I have many times done a complex sequence of actions only to realise a moment later when observing the consequences that my brain somehow took the wrong route!
A better for the user, but more complex (from software development point of view) solution would be:
Where possible communicate in advance what exact affect the action is going to make on the system (for instance Stack Overflow shows message preview above Post Your Answer button).
Give an immediate feedback to confirm once the action took place (SO highlights the freshly submitted answer, gmail displayes a confirmation when a message is sent etc).
Allow to undo or correct possible mistake (i.e. in SO case delete or edit the answer, Windows lets restore a file from recycle bin etc). For certain non-reversible actions it's still possible to give an undo capability but for a limited timeframe only (i.e. letting to cancel or change an online order during the first 10 minutes after its submission, or letting to recall an e-mail during the first 60 seconds after its been "sent", but actually queued in the outbox etc).
Sure, this is much more initial work than inserting a confimation message box, but instead of shifting the responsibility it attempts to solve the problem.
But if the OK/Cancels are not consistent, that might throw off or upset the user.
And don't do like some EULAs where a user is forced to scroll a panel to the bottom before the Agree button becomes clickable. Sometimes you just won't be able to get a user to read everything carefully.
If they really need to read it, maybe a short delay should happen before the buttons appear? This could also potentially be annoying to the user, but if it is a very critical question, it'd be worth it.
Edit: Or require some sort of additional mechanism than just clicking to "accept" the very important decision. A check box, key press, password, etc.
I recommend informing the user that this is a critical operation by using red text and explaining why is this an unsafe operation.
Also, rather than two buttons, have two radio buttons and one "Ok" button, with the "don't continue" radio button selected as default.
This will present the user with an uncommon interface, increasing cognitive load and slowing him down. Which is what you want here.
As always with anything with user interaction, you have a small space between helping the user and being annoying. I don't know you exact requirements but your idea seems OK(pun intended) to me.
It sounds like your user is going through a type of input wizard in the safety app.
Some ideas as alternatives to moving buttons.
Have a final screen to review all input before pressing the final ok.
Have a confirmation box after they hit ok explaining what the result of this action will be.
A disclaimer that require you to agree to it by checking a box before the user could continue.
Don't switch it around - you'll only confuse more than you'll help.
Instead, do like FireFox and not activate the control for 5 sec. - just make sure you include a timer or some sort of indicator that you're giving them a chance to read it over. If they click on it, it cuts off the timer, but requires they click one more time.
Don't know how much better it will be, but it could help.
Just remember, as the man said: You can't fix stupid.
This will give me headache. Especially when I accidentally close the application and forget to save my file :(
I see another good example of forcing user to "read" before click: Firefox always grayed out the button (a.k.a disable) the "OK" button. Therefore the user have to wait around 5 seconds before he can proceed to do anything. I think this is the best effort I have seen in forcing user to read (and think)
Another example I have seen is in "License and Agreements" page of the installer. Some of them required the user to scroll down to the end of the page before he/she can proceed to next step.
Keyboard shortcuts would still behave as before (and you'd be surprised how few people actually use mice (especially in LOB applications).
Vista (and OSX IIRC) have moved towards the idea of using specific verbs for each question (like the "Send"/"Don't send" when an app wants to crash and wants to submit a crashdump to MS)
In my opinion, I like the approach used by Outlook when an app tries to send an email via COM, with a timer before the buttons are allowed to be used (also affects keyboard shortcuts)
If you use Ok and Cancel as your interface you will always be allow the user to just skip your message or screen. If you then rearrange the Ok and Cancel you will just annoy your user.
A solution to this, if your goal is to insure the users understanding, is:
Question the user about the content. If you click Ok you are agreeing to Option 1, or if you click Ok you are agreeing to option 2. If they choose the correct answer, allow the action.
This will annoy the user, so if you can keep track of users, only do it to them once per message.
This is what I responded to Submit/Reset button order question and I think the same principle can be used here. The order does not really matter as far as you make sure the user can distinguish the two buttons. In the past what I have done is used a button for (submit/OK) button and used a link for (reset/cancel) button. The users can instantly tell that these two items are functionally different and hence treat them that way.
I am not really for OK/Cancel. It's overused and requires you to read the babbling in order to say what you are OKing or Canceling. Follow the idea of MacOSX UI: the button contains a simple, easy phrase that is meaningful by itself. Exampleç you change a file extension and a dialog pops up saying:
"Are you sure you want to change the extension from .py to .ps?"
If you perform the change, the document could be opened by a different application.
(Use .ps) (Keep .py)
It is way more communicative than OK/Cancel, and your question becomes almost superfluous, that is, you just need to keep active the rightmost button, which seems to be the standard.
As it concerns the raw question you posed. Never do it. Ever. Not even at gunpoint. Consistency is an important requisite for GUIs. If you are not consistent you will ruin the user experience, and your users will most likely to see this as a bug than a feature (indeed it would be a BUG). Consistency is very important. To break it, you must have very good reason, and there must not be another different, standard way to achieve the same effect.
I wonder if you're thinking about the option that exists in Visual Basic where you can set various prompts and response options; and one option is to allow you to switch Cancel and OK based on which should be the default; so the user could just hit enter and most of the time get the proper action.
If you really want to head in this direction (which I think is a bad idea, and I'm sure you will too after little reflection and reading all the oher posts) it would work even better to include a capcha display for OK.