I have an application, which displays me some data. I need to attach to this app's process, find the data I need in memory (one single number, actually), and save it somewhere. This application doesn't seem to use standard windows controls, so things aren't going to be as simple as reading controls data using AutoIt or something similar.
Currently I'm a self-learner database guy and have quite shallow knowledge about windows apps debugging. Not even sure if I asked my question correctly enough.
So, can you give me some starter guidelines about, say, what should I read first, and general directions I should work on?
Thanks.
To read memory of other application you need to open the process with respect of OpenProcess with at least PROCESS_VM_READ access rights and then use ReadProcessMemory to read any memory address from the process. If you are an administrator or have debug privilege you will be able to open any process with maximal access rights, you need only to enable SeDebugPrivilege before (see for example http://support.microsoft.com/kb/131065).
If you don't know a much about the memory of the destination process you can just enumerate the memory blocks with respect of VirtualQueryEx (see How does one use VirtualAllocEx do make room for a code cave? as an example where I examine the program code. The program data you can examine in the same way).
The most practical problem which I see is that you ask your question in too general way. If you explain more what kind of the data you are looking for I could probably suggest you a better way. For example if you could see the data somewhere you could examine the corresponding windows and controls with respect of Spy++ (a part of Visual Studio Tools). The most important are the class of windows (or controls) and the messages which will be send at the moment when the most interesting window are displayed. You can also use Process Monitor to trace all file and registry access at the time when the windows with the interesting information will be displayed. At least at the beginning you should examine the memory of the process with ReadProcessMemory at the moment when the data which you are looking for are displayed on the window.
If you will have no success in your investigations I'd recommend you to insert in your question more information.
My primary advice is: try to find any other method of integration than this. Even if you succeed, you'll be hostage to any kinds of changes in the target process, and possibly in the Windows O/S. What you are describing is behaviour most virus scanners should flag and hinder: if not now, then in the future.
That said, you can take a look at DLL injection. However, it sounds as if you're going to have to debug the heck out of the target process at the disassembly level: otherwise, how are you going to know what memory address to read?
I used to know the windows debugging API but it's long lost memory. How about using ollydbg:
http://www.ollydbg.de/
And controlling that with both ollydbg script and autoit?
Sounds interesting... but very difficult. Since you say this is a 'one-off', what about something like this instead?
Take a screenshot of this application.
Run the screenshot through an OCR program
If you are able to read the text you are looking for in a predictable way, you're halfway there!
So now if you can read a OCR'd screenshot of your application, it is a simple matter of writing a program that does the following:
Scripts the steps to get the data on the screen
Creates a screenshot of the data in question
Runs it through an OCR program like Microsoft Office Document Imaging
Extracts the relevant text and does 'whatever' with it.
I have done something like this before with pretty good results, but I would say it is a fragile solution. If the application changes, it stops working. If the OCR can't read the text, it stops working. If the OCR reads the wrong text, it might do worse things than stop working...
As the other posters have said, reaching into memory and pulling out data is a pretty advanced topic... kudos to you if you can figure out a way to do that!
I know this may not be a popular answer, due to the nature of what this software is used for, but programs like CheatEngine and ArtMoney allow you to search through all the memory reserved by a process for a given value, then refine the results till you find the address of the value you're looking for.
I learned this initially while trying to learn how to better protect my games after coming across a trainer for one of them, but have found the technique occasionally useful when debugging.
Here is an example of the technique described above in use: https://www.youtube.com/watch?v=Nv04gYx2jMw&t=265
Although NSIS allows you to build quite powerful installers, the "so low level language that it reminds me of assembly" that NSIS uses is quite prone to making mistakes and therefore, when you want your installer to do something more complex other than writing files, debugging is a must.
Until now I've used the following Dr Printf-like debugging technique:
In a .nsh file that I include everywhere, I define a NSIS_DEBUG_MSG macro according to the value of a DEBUG define
if DEBUG is on, the macro will trigger a MessageBox with the debug message
if DEBUG is off, the macro will do nothing
This method has served me well but it presents some disadvantages:
it requires me to fill the code that I feel it's the one failing with calls to NSIS_DEBUG_MSG and rebuild the installer several times until I get enough info to allow me to solve the problem
it will do me no good if my problem is that the installer itself fails (if the installer program dies)
So what I wanted to know is what debugging methods do you use for these installers so that hopefully I can improve mine.
During my time using NSIS, there is these things noticeable:
I have found out that nothing is more powerful than parsing !verbose 3 level output with self-made tool ;)
I have found out that you can NOT depend on any NSIS-based debugging method. It may crash .. and your installer will crash along with it. No pretty, eh'? :(
I have found out that enabling/disabling debugging on-demand is also very powerful weapon against idsses as it allows to distinguish between unstable and failed NSIS build ( it is easier to use CI terminology though... :) ).
I have found out that verbose output w/o realtime automated NSIS testing is like driving Cadillac with bicycle engine :)
Hope it helps for those accidentally visiting this question :)
EDIT: It's always good idea to start with 3rd party tools. For example, no need to hassle around GUI as it is always easier to use tools like:
EclipseNSIS ( I do not like it though :P )
NSIS Dialog Designer (http://nsis.sourceforge.net/NSIS_Dialog_Designer)
Self-made static code analyzer. I have made one for myself :P
EDIT #2: I have found out that pretty effective method for debugging is to use direct documentation automatization. Currently I use the following components:
http://nsis.sourceforge.net/NsScreenshot_plug-in ( See IMPORTANT NOTES to have no unexpected surprises... )
http://www.dokuwiki.org/cli#dwpagephp ( See this link for more information: http://www.microsoft.com/web/platform/phponwindows.aspx )
The result is that I got screenshot after nsDialog:Show plus I got updated images in wiki :) .. only stuff left is to fetch info from svnlook :)
EDIT #3: And the need of svnlook is also worked-around by building a svn log --xml exporting DLL using NSIS v2.44 header for Delphi and Lazarus IDE 0.9.30.2 :) Kudos to Lazarus!
Woohoo! :)
EDIT #4: Hit this small discussion here: http://forums.winamp.com/showthread.php?t=325521
What have saved me much time is to use the logs that are created by NSIS. Both the log while compiling the scripts, and the installation log. They allows me to check that the macros I have defined is in use, and that the installation actually run the scrips they should.
It might seem too little, but this is actually everything I need to keep my installation software of 50+ nsh files running, along with the divide an conquer principle.
You can download one of the special builds of NSIS from the official site that has advanced logging. This will give you very detailed logging information that makes debugging easier.
I use the DumpState plugin a lot, much better than a basic messagebox for stack issues. I alse tend to use a macro that sets all registers; $0=r0,$1=r1 etc so I know the stack is in the correct state. This is of course only useful during the design phase and not so much for debugging on the end users system...
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I've been a software developer for over twenty years, programming in C, Perl, SQL, Java, PHP, JavaScript, and recently Python. I've never had a problem I could not debug using some careful thought, and well-placed debugging print statements.
I respect that many people say that my techniques are primitive, and using a real debugger in an IDE is much better. Yet from my observation, IDE users don't appear to debug faster or more successfully than I can, using my stone knives and bear skins. I'm sincerely open to learning the right tools, I've just never been shown a compelling advantage to using visual debuggers.
Moreover, I have never read a tutorial or book that showed how to debug effectively using an IDE, beyond the basics of how to set breakpoints and display the contents of variables.
What am I missing? What makes IDE debugging tools so much more effective than thoughtful use of diagnostic print statements?
Can you suggest resources (tutorials, books, screencasts) that show the finer techniques of IDE debugging?
Sweet answers! Thanks much to everyone for taking the time. Very illuminating. I voted up many, and voted none down.
Some notable points:
Debuggers can help me do ad hoc inspection or alteration of variables, code, or any other aspect of the runtime environment, whereas manual debugging requires me to stop, edit, and re-execute the application (possibly requiring recompilation).
Debuggers can attach to a running process or use a crash dump, whereas with manual debugging, "steps to reproduce" a defect are necessary.
Debuggers can display complex data structures, multi-threaded environments, or full runtime stacks easily and in a more readable manner.
Debuggers offer many ways to reduce the time and repetitive work to do almost any debugging tasks.
Visual debuggers and console debuggers are both useful, and have many features in common.
A visual debugger integrated into an IDE also gives you convenient access to smart editing and all the other features of the IDE, in a single integrated development environment (hence the name).
Some examples of some abilities that an IDE debugger will give you over trace messages in code:
View the call stack at any point in time, giving you a context for your current stack frame.
Step into libraries that you are not able to re-compile for the purposes of adding traces (assuming you have access to the debug symbols)
Change variable values while the program is running
Edit and continue - the ability to change code while it is running and immediately see the results of the change
Be able to watch variables, seeing when they change
Be able to skip or repeat sections of code, to see how the code will perform. This allows you to test out theoretical changes before making them.
Examine memory contents in real-time
Alert you when certain exceptions are thrown, even if they are handled by the application.
Conditional breakpointing; stopping the application only in exceptional circumstances to allow you to analyse the stack and variables.
View the thread context in multi-threaded applications, which can be difficult to achieve with tracing (as the traces from different threads will be interleaved in the output).
In summary, print statements are (generally) static and you'll need to re-compile to get additional information if your original statements weren't detailed enough. The IDE removes this static barrier, giving you a dynamic toolkit at your fingertips.
When I first started coding, I couldn't understand what the big deal with debuggers was and I thought I could achieve anything with tracing (granted, that was on unix and the debugger was GDB). But once you learn how to properly use a graphical debugger, you don't want to go back to print statements.
An IDE debugger lets you change the
values of variables at run-time.
An IDE
debugger lets you see the value of
variables you didn't know you wanted
to see when execution began.
An IDE
debugger lets you see the call stack
and examine the state of the
function passed weird values.
(think this function is called from
hundreds of places, you don't know
where these weird values are coming
from)
An IDE debugger lets you
conditionally break execution at any
point in code, based on a condition,
not a line number.
An IDE debugger will let you examine the state of the program in the case of an unhandled exception instead of just crapping out.
Here's one thing that you definitely cannot debug with "print" statement, which is when a customer brings you memory dump and says "your program crashed, can you tell me why?"
Print statements all through your code reduces readability.
Adding and removing them for debug purposes only is time consuming
Debuggers track the call stack making it easy to see where you are
Variables can be modified on the fly
Adhoc commands can be executed during a pause in execution to assist diagnosing
Can be used IN CONJUNCTION with print statements : Debug.Write("...")
I think debugging using print statements is a lost art, and very important for every developer to learn. Once you know how to do that, certain classes of bugs become much easier to debug that way than through an IDE. Programmers who know this technique also have a really good feel of what's useful information to put in a log message (not to mention you'll actually end up reading the log) for non-debugging purposes as well.
That said, you really should know how to use the step-through debugger, since for a different class of bugs it is WAY easier. I'll leave it up to the other excellent answers already posted to explain why :)
Off the top of my head:
Debugging complex objects - Debuggers allow you to step deep into an object's innards. If your object has, say, an array of array of complex objects, print statements will only get you so far.
The ability to step past code - Debuggers will also allow you to skip past code you don't want to execute. True, you could do this manually as well, but it's that much more code you have to inject.
As alternative to debug in IDE you can try great Google Chrome extension PHP Console with php library that allows to:
See errors & exception in Chrome JavaScript console & in notification popups.
Dump any type variable.
Execute PHP code remotely.
Protect access by password.
Group console logs by request.
Jump to error file:line in your text editor.
Copy error/debug data to clipboard (for testers).
I haven't been developing for nearly 20 years, but I find that using a IDE / debugger I can :
see all kinds of things I might not have thought to have included in a print statement
step through code to see if it matches the path I thought it would take
set variables to certain values to make code take certain branches
One reason to use the IDE might be that modern IDEs support more than simple breakpoints. For example, Visual Studio offers the following advanced debugging features:
define conditional breakpoints (break only if a condition is met, or only on the n-th time the statement at the breakpoint is executed)
break on an unhandled exception or whenever a (specific) ecxeption is to be thrown
change variable while debugging
repeating a piece of code by setting the next line to be executed
etc.
Also, when using the debugger, you won't have to remove all your print statements once you have finished debugging.
In my experience, simple printouts have one huge advantage that no one seems to mention.
The problem with an IDE debugger is that everything happens at real time. You halt the program at a certain time, then you step through the steps one at a time and it is impossible to go back if you suddenly want to see what happened before. This is completley at odds with how our brain works. The brain collects information, and gradually forms an oppinion. It might be necessary to iterate the events several times in doing so, but once you have stepped past a certain point, you cannot go back.
In contrast to this, a selected series of printouts/logging gives you a "spatial projection of the temporal events". It gives you a complete story of what happened, and you can go back and fourth several times very easily by just scrolling up and down. It makes it easy to answer questions like "did A occur before B happened". It can make you see patterns you wernt even looking for.
So in my experience. IDE and debuggers are fantastic tools to solve simple problems when something in one single call-stack went wrong, and explore the current state of the machine at a certain crash.
However, when we approach more difficoult problems where gradual changing of state is involved. Where for example one algorithm corrupted a data structure, that in turn caused anohter algorithm to fail. Or if we want to answer questions like "how often do this happen", "do things happen in the order and in the way as I imagine them to happen". etc. Then the "old fashined" logging/printout technique has a clear advantage.
The best things is to use either technique when it is most suitable, for example use logging/printouts to get to some bugs, and pause at a breakpoint where we need to explore the current state more in detail.
There are also hybrid approaches. For example, when you do console.log(object) you get a data-structure widget in the log that you can expand and explore more in detail.This is many times a clear advantage over a "dead" text log.
One thing that I'm surprised I haven't seen in another answer is that the 2 debugging methods are not mutually exclusive.
printf debugging can work quite nicely even if you're using a standard debugger (whether IDE based or not). In particular with a logging framework so you can leave all or most of in the released product to help with diagnosing customer problems.
As noted in pretty much all the other answers here, the key nice thing about a standard debugger is that it allows you to more easily examine (and potentially change) the details of the program state. You don't have to know up front what you might want to look at - it's all available at your fingertips (more or less).
Because debugging multi-threaded applications with print statements will drive you bananas. Yes you can still do it with print statements but you'd need a lot of them and unravelling the sequential print out of statements to emulate the multi-threaded executiong would take a long long time.
Human brains are only single-threaded unfortunately.
Since you asked for pointers to books... As far as Windows debugging goes, John Robbins has several editions of a good book on Windows debugging:
Debugging Applications for Microsoft .NET and Microsoft Windows
Note that the most recent edition (Debugging Microsoft .NET 2.0 Applications) is .NET only, so you might want an older one (like in the first link) if you want native code debugging (it covers both .NET and native).
I personally feel the answer is as simple as "A integrated debugger/IDE gives you a wealth of different information quickly without the need for punching in commands. The information tends to be there in front of you without you haven't tell it what to show you.
The ease in which the information can be retrieved is what makes them better than just command-line debugging, or "printf" debugging.
Advantages of a debugger over a printf (note not an IDE debugger but any debugger)
Can set watchpoints.
This is one of my favourite ways of finding memory corruptions
Can debug a binary that you can't recompile at the moment
Can debug a binary that takes a long time to recompile
Can change variables on the fly
Can call functions on the fly
Doesn't have the problem where debug statemenets are not flushed and hence timing issue can not be debugged acuratly
Debuggers help with core dumps, print statements dont'
This is what I use most on VS.NET debugging windows:
Call stack, which is also a great way to figure out someone else's code
Locals & Watches.
Immediate window, which is basically a C# console and also lets me change variable contents, initialize stuff etc.
The ability to skip a line, set the next statement to be executed somewhere else.
The ability to hover over variables and have a tool-tip showing me their values.
In summary, it gives me a 360 degree view of the state of my executing code, not just a small window.
Never found a book teaching this kind of stuff, but then again, it seems to be quite simple, it's pretty much WYSIWYG.
A debugger can attach to a running process
Often easier to debug threaded code from a debugger
With an IDE debugger you can see the values of ALL the variables in the current scope (all the way up the call stack) whenever you halt execution.
Print statements can be great but dumping so much information to the screen at any given place can produce a whole lot of print statements.
Also, many IDE debuggers let you type in and evaluate methods, and evaluate members while you are halted, which further increases the amount of print statements you'd have to do.
I do feel that debuggers are better for some languages than for others however...
My general opinion is that IDE debuggers are absolutely, amazingly wonderful for managed languages like Java or C#, are fairly useful for C++, and are not very useful for scripting languages like Python (but it could be that I just haven't tried a good debugger for any scripting languages yet).
I absolutely love the debugger in IntelliJ IDEA when I do Java development. I just use print statements when I use Python.
As someone said above: Debugger != IDE.
gdb and (back in the day) TurboDebugger (stand-alone) work just fine for the languages they support[ed], thank you. (or an even older technology: Clipper debugger linked into the xBase executable itself) -- none of these required an IDE
Also, though C/++ coding is more rare, printf statements sometimes mask off the very bug you are trying to find! (initialization problems in auto vars on the stack, for instance, or memory allocation/alignment)
Finally, as others stated, you can use both. Some real-time-ish problems almost require a print, or at least a judicious "*video_dbg = ( is_good ? '+' : '-');" somewhere into video memory. My age is showing, this was under DOS :-)
TMTOWTDI
In addition to much of what the other posters have said, I really like stepping through one line at a time along with the computer, as it forces me to think about one line at a time. Often I will catch the bug without even looking at variable values simply because I am forced to look at it as I click the 'next line' button. However, I don't think my answer will help you, Bill, because you probably have this skill already.
As far as learning resources go, I haven't used any -- I just explore all the menus and options.
Is this even real question from real programmer?
Anyone who spent even 5 mins debugging with print statements and debugging with IDE - it will OCCUR to him/her without even asking!
I've used both prints and IDEs for debugging and I would much rather debug using an IDE. The only time for me when that doesn't work is in time critical situations (like debugging online games) where you litter the code with print statements and then look at the log files after it has gone horribly wrong. Then if you still cannot figure it out, add more prints and repeat.
Just wanted to mention a useful feature of a console debugger vs printf and vs debugger in an IDE.
You can attach to a remote application (obvioustly, compiled in DEBUG mode) and inspect its state dumping the debugger output to a file using POSIX tee utility. Compared to printf, you can choose where to output the state in run-time.
It helped me a lot when I was debugging Adobe Flash applications deployed in an agressive environment. You just need to define some actions that print required state in each breakpoint, start the console debugger with fdb | tee output.log, and walk through some breakpoints. After that you can print the log and analyse the information by thorough comparison of the state in different breakpoints.
Unfortunatelly, this feature [logging to a file] is rarely available in GUI debuggers, making developers compare the state of objects in their head.
By the way, my opinion is that one should plan where and what to debug before staring a debugger.
Well another thing is that if you join a new old project and nobody really knows how the code is doing what it's doing, then you can't debug by echoing variables/objects/... b/c you have no idea what code is executed at all.
At my job I am facing exactly that kind of situation and visual XDebuging helps me getting an idea about what is going on and where, at all.
Best regards
Raffael
In addition to the many things that have been already mentioned, one of the most important advantages of a debugger over printf is that using printf statements assumes that you know in which function the bug resides. In many cases you don't, so you have to make a few guesses and add print statements to many other functions in order to localise it. The bug may be in framework code or somewhere far removed from where you think it is. In a debugger it is far easier to set breakpoints to examine the state in different areas of the code and at different points in time.
Also, a decent debugger will let you do printf-style debugging by attaching conditions and actions to breakpoints, so that you still retain the benefits of printf debugging, but without modifying the code.
Debugging in an IDE is invaluable in an environment where error logs and shell access are unavailable, such as a shared host. In that case, an IDE with a remote debugger is the only tool which allows you to do simple things such as view stderr or stdout.
A problem with using print statements is it makes a mess of your code. IE, you have a function with 10 parts to it and you know it crashes somewhere, but you're not sure where. So you add in 10 extra print statements to pinpoint where the bug is. Once you've found and solved your bug, you now have to clean up by removing all of those print statements. Maybe you'll do that. Maybe you'll forget and it'll end up in production and your user's console will be full of debug prints.
Wauw, do I like this question. I never dared to pose it...
It seems that people just have different ways of working.
For me what works best is:
Having a solid mind model of my code, including memory management
Using instrumentation (like print statements) to follow what's happening.
I've earned my living programming for over 40 years now, working at non-trivial technical and scientific applications in C++ and Python daily, and I have the personal experience that a debugger doesn't help me a bit.
I don't say that's good. I don't say that's bad. I just want to share it.
It's not just debugging. An IDE helps you build better software faster in a lot of ways:
refactoring tools
intellisense to make api's more discoverable, or remind of exact spelling/case of familiar items(not much use if you've used the same system for 15 years, but that's rare)
save on typing by autocompleting variable and class names
find certain kinds of errors before you even start to compile
Automatically jump to variable/method/class declarations/definitions, even if they're not in the same file or folder.
Break on unhandled and handled exceptions
I could go on.
We wish to provide people to be able to add some logic to their accounts(say, given a few arguments, how to compute a particular result). So, essentially, this would be tantamount to writing simple business rules with support for conditionals and expressions. However, the challenge is to provide them a simple online editor where they can create the logic (preferably) by completely visual means (drag/drop Expr-tree nodes maybe -- kinda like Y! pipes).
Does anybody know of a scripting/macro/domain-specific language that lets people do this? The challenge is the visual editor, since we don't wish to invest in developing the UI to do the editing. The basic requirements would be:
1. Embedded into another language, or run securely (no reboot -n or <JUNK-DANGEROUS-COMMAND> >> ~/.bashrc)
2. Easily accessible to users without coding background (no need of any advanced features)
3. Preferably have a simple GUI based editor to create the logic programs accessible to non-developers (kinda like spreadsheets)
4. Some ability to generate compile-time warnings (invalid code) would be good (Type safety?)
5. Ability to embed some data before execution which is available to the interpreter (Eg., name, birthday, amount)
Anybody tried doing something like this and got any ideas? I looked at Lua, Io, Python, Ruby and a host of others, but the challenge essentially is that I don't think non-programmers will be able to understand the code all that much. Something that could be added via "meta-programming" to say a Ruby would be good as well, if an editor could be easily developed!
As a matter fact, Microsoft is developing Oslo, which is right up your alley.
Chris Sells has been writing a lot about it recently.
It is designed to be a way to author DSLs and also to visually author these models with a graphical tool called Quadrant. Sounds very very similar to what you are looking for.
Open source wise, Ruby I think can be close, as you can see if you look at _whytheluckystiff's Try Ruby or Hackety.
I don't think you'll find anything that isn't too generic, especially regarding the GUI editor. There's no generic tools as far as I know that will be able to automatically interface with your program and be able to query data from it and interpret the script into commands in your software -- if there is I'd like to have a copy. Not being flippant, but you will have to do some (probably alot) of work to get this working. It will probably result in you writing a custom DSL.
I would take a look at PowerShell. You could surface all the activities a user would like to script in a very readable way.
There is some talk of using PowerShell to create a DSL on the PowerShell team blog and Bruce Payette, the technical lead, talks about this in his book Windows PowerShell in Action from Manning.
At the other end of the scale is to write something simple as a HyperText Application (HTA) -- assuming Windows of course -- along the lines of my Clive tool. The article on the blog doesn't mention the HTA version, but essentially I could enter VBScript-ish code into one textarea and interpret it on the spot, output going into another text area on the form.
With HTAs giving you all the form control of HTML, plus the DOM, you could come up with something interesting fairly quickly.