Related
As a developer and constant user of minipfoler, I use stakoverflow as the benchmark for my .NET sites. That is because the entire stack network is just a blazingly fast.
I know miniprofiler is used on stackexchange. There is a whole developers thing that can be used on stack but can we enable the stats to see how fast it really is?
I might be bit over obsessive here - but I am looking to improve permanences in milliseconds and the only viable benchmark is a large and complex site like stack exchange.
I know it might be a security issue to see live data but I just really want a benchmark (screenshot / guidelines) to see how far I can optimize my .NET MVC web application.
My actual IIS and MVC performance is fantastic and I think I am more concerned about server replies and client side stuff. So can I (and should I) put more effort into smashing down this response time?
This site is hosted in Azure Cloupapp and using Azure DB - I know about 60~180ms is used on connection times that are out of my control.
How can I improve times between Paint, Load and Complete?
I find that I answer my own question on StackExchange more often now a days. Not sure what that means. But this in interesting what I found while dealing with other Q&A's (And it answered this question)
Yes, you should avoid the obvious beginner mistakes of string
concatenation, the stuff every programmer learns their first year on
the job. But after that, you should be more worried about the
maintainability and readability of your code than its performance. And
that is perhaps the most tragic thing about letting yourself get
sucked into micro-optimization theater -- it distracts you from your
real goal: writing better code.
Posted by Jeff Atwood
There is no real problem in performance or serious delays. Its just an obsession that wont lead to much satisfaction.
The 'dudes' got a point. As long as my code is readable and it runs fast - what the heck more do I want?
PERFECTION! - Waste of time, lol#me!
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I've been a software developer for over twenty years, programming in C, Perl, SQL, Java, PHP, JavaScript, and recently Python. I've never had a problem I could not debug using some careful thought, and well-placed debugging print statements.
I respect that many people say that my techniques are primitive, and using a real debugger in an IDE is much better. Yet from my observation, IDE users don't appear to debug faster or more successfully than I can, using my stone knives and bear skins. I'm sincerely open to learning the right tools, I've just never been shown a compelling advantage to using visual debuggers.
Moreover, I have never read a tutorial or book that showed how to debug effectively using an IDE, beyond the basics of how to set breakpoints and display the contents of variables.
What am I missing? What makes IDE debugging tools so much more effective than thoughtful use of diagnostic print statements?
Can you suggest resources (tutorials, books, screencasts) that show the finer techniques of IDE debugging?
Sweet answers! Thanks much to everyone for taking the time. Very illuminating. I voted up many, and voted none down.
Some notable points:
Debuggers can help me do ad hoc inspection or alteration of variables, code, or any other aspect of the runtime environment, whereas manual debugging requires me to stop, edit, and re-execute the application (possibly requiring recompilation).
Debuggers can attach to a running process or use a crash dump, whereas with manual debugging, "steps to reproduce" a defect are necessary.
Debuggers can display complex data structures, multi-threaded environments, or full runtime stacks easily and in a more readable manner.
Debuggers offer many ways to reduce the time and repetitive work to do almost any debugging tasks.
Visual debuggers and console debuggers are both useful, and have many features in common.
A visual debugger integrated into an IDE also gives you convenient access to smart editing and all the other features of the IDE, in a single integrated development environment (hence the name).
Some examples of some abilities that an IDE debugger will give you over trace messages in code:
View the call stack at any point in time, giving you a context for your current stack frame.
Step into libraries that you are not able to re-compile for the purposes of adding traces (assuming you have access to the debug symbols)
Change variable values while the program is running
Edit and continue - the ability to change code while it is running and immediately see the results of the change
Be able to watch variables, seeing when they change
Be able to skip or repeat sections of code, to see how the code will perform. This allows you to test out theoretical changes before making them.
Examine memory contents in real-time
Alert you when certain exceptions are thrown, even if they are handled by the application.
Conditional breakpointing; stopping the application only in exceptional circumstances to allow you to analyse the stack and variables.
View the thread context in multi-threaded applications, which can be difficult to achieve with tracing (as the traces from different threads will be interleaved in the output).
In summary, print statements are (generally) static and you'll need to re-compile to get additional information if your original statements weren't detailed enough. The IDE removes this static barrier, giving you a dynamic toolkit at your fingertips.
When I first started coding, I couldn't understand what the big deal with debuggers was and I thought I could achieve anything with tracing (granted, that was on unix and the debugger was GDB). But once you learn how to properly use a graphical debugger, you don't want to go back to print statements.
An IDE debugger lets you change the
values of variables at run-time.
An IDE
debugger lets you see the value of
variables you didn't know you wanted
to see when execution began.
An IDE
debugger lets you see the call stack
and examine the state of the
function passed weird values.
(think this function is called from
hundreds of places, you don't know
where these weird values are coming
from)
An IDE debugger lets you
conditionally break execution at any
point in code, based on a condition,
not a line number.
An IDE debugger will let you examine the state of the program in the case of an unhandled exception instead of just crapping out.
Here's one thing that you definitely cannot debug with "print" statement, which is when a customer brings you memory dump and says "your program crashed, can you tell me why?"
Print statements all through your code reduces readability.
Adding and removing them for debug purposes only is time consuming
Debuggers track the call stack making it easy to see where you are
Variables can be modified on the fly
Adhoc commands can be executed during a pause in execution to assist diagnosing
Can be used IN CONJUNCTION with print statements : Debug.Write("...")
I think debugging using print statements is a lost art, and very important for every developer to learn. Once you know how to do that, certain classes of bugs become much easier to debug that way than through an IDE. Programmers who know this technique also have a really good feel of what's useful information to put in a log message (not to mention you'll actually end up reading the log) for non-debugging purposes as well.
That said, you really should know how to use the step-through debugger, since for a different class of bugs it is WAY easier. I'll leave it up to the other excellent answers already posted to explain why :)
Off the top of my head:
Debugging complex objects - Debuggers allow you to step deep into an object's innards. If your object has, say, an array of array of complex objects, print statements will only get you so far.
The ability to step past code - Debuggers will also allow you to skip past code you don't want to execute. True, you could do this manually as well, but it's that much more code you have to inject.
As alternative to debug in IDE you can try great Google Chrome extension PHP Console with php library that allows to:
See errors & exception in Chrome JavaScript console & in notification popups.
Dump any type variable.
Execute PHP code remotely.
Protect access by password.
Group console logs by request.
Jump to error file:line in your text editor.
Copy error/debug data to clipboard (for testers).
I haven't been developing for nearly 20 years, but I find that using a IDE / debugger I can :
see all kinds of things I might not have thought to have included in a print statement
step through code to see if it matches the path I thought it would take
set variables to certain values to make code take certain branches
One reason to use the IDE might be that modern IDEs support more than simple breakpoints. For example, Visual Studio offers the following advanced debugging features:
define conditional breakpoints (break only if a condition is met, or only on the n-th time the statement at the breakpoint is executed)
break on an unhandled exception or whenever a (specific) ecxeption is to be thrown
change variable while debugging
repeating a piece of code by setting the next line to be executed
etc.
Also, when using the debugger, you won't have to remove all your print statements once you have finished debugging.
In my experience, simple printouts have one huge advantage that no one seems to mention.
The problem with an IDE debugger is that everything happens at real time. You halt the program at a certain time, then you step through the steps one at a time and it is impossible to go back if you suddenly want to see what happened before. This is completley at odds with how our brain works. The brain collects information, and gradually forms an oppinion. It might be necessary to iterate the events several times in doing so, but once you have stepped past a certain point, you cannot go back.
In contrast to this, a selected series of printouts/logging gives you a "spatial projection of the temporal events". It gives you a complete story of what happened, and you can go back and fourth several times very easily by just scrolling up and down. It makes it easy to answer questions like "did A occur before B happened". It can make you see patterns you wernt even looking for.
So in my experience. IDE and debuggers are fantastic tools to solve simple problems when something in one single call-stack went wrong, and explore the current state of the machine at a certain crash.
However, when we approach more difficoult problems where gradual changing of state is involved. Where for example one algorithm corrupted a data structure, that in turn caused anohter algorithm to fail. Or if we want to answer questions like "how often do this happen", "do things happen in the order and in the way as I imagine them to happen". etc. Then the "old fashined" logging/printout technique has a clear advantage.
The best things is to use either technique when it is most suitable, for example use logging/printouts to get to some bugs, and pause at a breakpoint where we need to explore the current state more in detail.
There are also hybrid approaches. For example, when you do console.log(object) you get a data-structure widget in the log that you can expand and explore more in detail.This is many times a clear advantage over a "dead" text log.
One thing that I'm surprised I haven't seen in another answer is that the 2 debugging methods are not mutually exclusive.
printf debugging can work quite nicely even if you're using a standard debugger (whether IDE based or not). In particular with a logging framework so you can leave all or most of in the released product to help with diagnosing customer problems.
As noted in pretty much all the other answers here, the key nice thing about a standard debugger is that it allows you to more easily examine (and potentially change) the details of the program state. You don't have to know up front what you might want to look at - it's all available at your fingertips (more or less).
Because debugging multi-threaded applications with print statements will drive you bananas. Yes you can still do it with print statements but you'd need a lot of them and unravelling the sequential print out of statements to emulate the multi-threaded executiong would take a long long time.
Human brains are only single-threaded unfortunately.
Since you asked for pointers to books... As far as Windows debugging goes, John Robbins has several editions of a good book on Windows debugging:
Debugging Applications for Microsoft .NET and Microsoft Windows
Note that the most recent edition (Debugging Microsoft .NET 2.0 Applications) is .NET only, so you might want an older one (like in the first link) if you want native code debugging (it covers both .NET and native).
I personally feel the answer is as simple as "A integrated debugger/IDE gives you a wealth of different information quickly without the need for punching in commands. The information tends to be there in front of you without you haven't tell it what to show you.
The ease in which the information can be retrieved is what makes them better than just command-line debugging, or "printf" debugging.
Advantages of a debugger over a printf (note not an IDE debugger but any debugger)
Can set watchpoints.
This is one of my favourite ways of finding memory corruptions
Can debug a binary that you can't recompile at the moment
Can debug a binary that takes a long time to recompile
Can change variables on the fly
Can call functions on the fly
Doesn't have the problem where debug statemenets are not flushed and hence timing issue can not be debugged acuratly
Debuggers help with core dumps, print statements dont'
This is what I use most on VS.NET debugging windows:
Call stack, which is also a great way to figure out someone else's code
Locals & Watches.
Immediate window, which is basically a C# console and also lets me change variable contents, initialize stuff etc.
The ability to skip a line, set the next statement to be executed somewhere else.
The ability to hover over variables and have a tool-tip showing me their values.
In summary, it gives me a 360 degree view of the state of my executing code, not just a small window.
Never found a book teaching this kind of stuff, but then again, it seems to be quite simple, it's pretty much WYSIWYG.
A debugger can attach to a running process
Often easier to debug threaded code from a debugger
With an IDE debugger you can see the values of ALL the variables in the current scope (all the way up the call stack) whenever you halt execution.
Print statements can be great but dumping so much information to the screen at any given place can produce a whole lot of print statements.
Also, many IDE debuggers let you type in and evaluate methods, and evaluate members while you are halted, which further increases the amount of print statements you'd have to do.
I do feel that debuggers are better for some languages than for others however...
My general opinion is that IDE debuggers are absolutely, amazingly wonderful for managed languages like Java or C#, are fairly useful for C++, and are not very useful for scripting languages like Python (but it could be that I just haven't tried a good debugger for any scripting languages yet).
I absolutely love the debugger in IntelliJ IDEA when I do Java development. I just use print statements when I use Python.
As someone said above: Debugger != IDE.
gdb and (back in the day) TurboDebugger (stand-alone) work just fine for the languages they support[ed], thank you. (or an even older technology: Clipper debugger linked into the xBase executable itself) -- none of these required an IDE
Also, though C/++ coding is more rare, printf statements sometimes mask off the very bug you are trying to find! (initialization problems in auto vars on the stack, for instance, or memory allocation/alignment)
Finally, as others stated, you can use both. Some real-time-ish problems almost require a print, or at least a judicious "*video_dbg = ( is_good ? '+' : '-');" somewhere into video memory. My age is showing, this was under DOS :-)
TMTOWTDI
In addition to much of what the other posters have said, I really like stepping through one line at a time along with the computer, as it forces me to think about one line at a time. Often I will catch the bug without even looking at variable values simply because I am forced to look at it as I click the 'next line' button. However, I don't think my answer will help you, Bill, because you probably have this skill already.
As far as learning resources go, I haven't used any -- I just explore all the menus and options.
Is this even real question from real programmer?
Anyone who spent even 5 mins debugging with print statements and debugging with IDE - it will OCCUR to him/her without even asking!
I've used both prints and IDEs for debugging and I would much rather debug using an IDE. The only time for me when that doesn't work is in time critical situations (like debugging online games) where you litter the code with print statements and then look at the log files after it has gone horribly wrong. Then if you still cannot figure it out, add more prints and repeat.
Just wanted to mention a useful feature of a console debugger vs printf and vs debugger in an IDE.
You can attach to a remote application (obvioustly, compiled in DEBUG mode) and inspect its state dumping the debugger output to a file using POSIX tee utility. Compared to printf, you can choose where to output the state in run-time.
It helped me a lot when I was debugging Adobe Flash applications deployed in an agressive environment. You just need to define some actions that print required state in each breakpoint, start the console debugger with fdb | tee output.log, and walk through some breakpoints. After that you can print the log and analyse the information by thorough comparison of the state in different breakpoints.
Unfortunatelly, this feature [logging to a file] is rarely available in GUI debuggers, making developers compare the state of objects in their head.
By the way, my opinion is that one should plan where and what to debug before staring a debugger.
Well another thing is that if you join a new old project and nobody really knows how the code is doing what it's doing, then you can't debug by echoing variables/objects/... b/c you have no idea what code is executed at all.
At my job I am facing exactly that kind of situation and visual XDebuging helps me getting an idea about what is going on and where, at all.
Best regards
Raffael
In addition to the many things that have been already mentioned, one of the most important advantages of a debugger over printf is that using printf statements assumes that you know in which function the bug resides. In many cases you don't, so you have to make a few guesses and add print statements to many other functions in order to localise it. The bug may be in framework code or somewhere far removed from where you think it is. In a debugger it is far easier to set breakpoints to examine the state in different areas of the code and at different points in time.
Also, a decent debugger will let you do printf-style debugging by attaching conditions and actions to breakpoints, so that you still retain the benefits of printf debugging, but without modifying the code.
Debugging in an IDE is invaluable in an environment where error logs and shell access are unavailable, such as a shared host. In that case, an IDE with a remote debugger is the only tool which allows you to do simple things such as view stderr or stdout.
A problem with using print statements is it makes a mess of your code. IE, you have a function with 10 parts to it and you know it crashes somewhere, but you're not sure where. So you add in 10 extra print statements to pinpoint where the bug is. Once you've found and solved your bug, you now have to clean up by removing all of those print statements. Maybe you'll do that. Maybe you'll forget and it'll end up in production and your user's console will be full of debug prints.
Wauw, do I like this question. I never dared to pose it...
It seems that people just have different ways of working.
For me what works best is:
Having a solid mind model of my code, including memory management
Using instrumentation (like print statements) to follow what's happening.
I've earned my living programming for over 40 years now, working at non-trivial technical and scientific applications in C++ and Python daily, and I have the personal experience that a debugger doesn't help me a bit.
I don't say that's good. I don't say that's bad. I just want to share it.
It's not just debugging. An IDE helps you build better software faster in a lot of ways:
refactoring tools
intellisense to make api's more discoverable, or remind of exact spelling/case of familiar items(not much use if you've used the same system for 15 years, but that's rare)
save on typing by autocompleting variable and class names
find certain kinds of errors before you even start to compile
Automatically jump to variable/method/class declarations/definitions, even if they're not in the same file or folder.
Break on unhandled and handled exceptions
I could go on.
By 'challenging development environment' I don't mean you're on a small boat that's rocking up and down and someone is holding a gun to your head. I mean, are the tools at your disposal making the problem difficult?
Development is typically a cycle of code, run, observe the result, repeat. In some environments this is a very quick and painless process, but in others it's very difficult. We end up using little tricks to help us observe the result and run the code faster.
I was thinking of this because I just started using SSIS (an ETL tool included with SQL Server 2005/8). It's been quite challenging for me to make progress, mainly because there's no guidance on what all the dialogs mean and also because the errors are very cryptic and most of the time don't really tell you what the problem is.
I think the easiest environment I've had to work in was VB6 because there you can edit code while the application is running and it will continue running with your new code! You don't even have to run it again. This can save you a lot of time. Surprisingly, in Netbeans with Java code, you can do the same. It steps out of the method and re-runs the method with the new code.
In SQL Server 2000 when there is an error in a trigger you get no stack trace, which can make it really tricky to locate where the problem occurred since an insert can have a cascading effect and trigger many triggers. In Oracle you get a very nice little stack trace with line numbers so resolving the problem is very easy.
Some of the things that I see really help in locating problems:
Good error messages when things go wrong.
Providing a stack trace when a problem occurs.
Debug environment where you can pause, then output the value of variables and step to follow the execution path.
A graphical debug environment that shows the code as it's running.
A screen that can show the current values of variables so you can print to them.
Ability to turn on debug logging on a production system.
What is the worst you've seen and what can be done to get around these limitations?
EDIT
I really didn't intend for this to be flame bait. I'm really just looking for ideas to improve systems so that if I'm creating something I'll think about these things and not contribute to people's problems. I'm also looking for creative ways around these limitations that I can use if I find myself in this position.
I was working on making modifications to Magento for a client. There is very little information on how the Magento system is organized. There are hundreds of folders and files, and there are at least a thousand view files. There was little support available from Magento forums, and I suspect the main reason for this lack of information is because the creators of Magento want you to pay them to become a certified Magento developer. Also, at that time last year there was no StackOverflow :)
My first task was to figure out how the database schema worked and which table stored some attributes I was looking for. There are over 300 tables in Magento, and I couldn't find out how the SQL queries were being done. So I had just one option...
I exported the entire database (300+ tables, and at least 20,000 lines of SQL code) into a .sql file using PhpMyAdmin, and I 'committed' this file into the subversion repositry. Then, I made some changes to the database using the Magento administration panel, and redownloaded the .sql file. Then, I ran a DIFF using TortioseSvn, and scrolled through the 20k+ lines file to find which lines had changed, LOL. As crazy as it sounds, it did work, and I was able to figure out which tables I needed to access.
My 2nd problem was, because of the crazy directory structure, I had to ftp to about 3 folders at the same time for trivial changes. So I had to keep 3 windows of my ftp program open, switch between them and ftp each time.
The 3rd problem was figuring out how the url mapping worked and where some of the code I wanted was being stored. Here, by sheer luck, I managed to find the Model class I was looking for.
Mostly by sheer luck and other similar crazy adventures I managed to work my way through and complete the project. Since then, StackOverflow was started and by a helpful answer to this bounty question I was able to finally get enough information about Magento that I can do future projects in a less crazy manner (hopefully).
Try keypunching your card deck in Fortran, complete with IBM JCL (Job Control Language), handing it in at the data center window, coming back the next morning and getting an inch-thick stack of printer paper with the hex dump of your crash, and a list of the charges to your account.
Grows hair on your fingernails.
I guess that was an improvement on the prior method of sitting at the console, toggling switches and reading the lights.
Occam on a 400x transputer network. As there was only one transputer that could output to console debugging was a nightmare. Had to build a test harness on a Sun network.
I took a class once, that was loosely based on SICP, except it was taught in Dylan rather than Scheme. Actually, it was in the old Dylan syntax, the prefix one that was based on Scheme. But because there were no interpreters for that old version of Dylan, the professor wrote one. In Java. As an applet. Which meant that it had no access to the filesystem; you had to write all of your code in a separate text editor, and then paste it into the Dylan interpreter. Oh, and it had no debugging facilities, of course. And being a Dylan interpreter written in Java, and this was back in 2000, it was ungodly slow.
Print statement debugging, lots of copying and pasting, and an awful lot of cursing at the interpreter were involved.
Back in the 90's, I was developing applications in Clipper, a compilable dBase-like language. I don't remember if it came with a debugger, we often used a 3rd-party debugger called 'Mr Debug' (really!). Although Clipper was fast, some of our more intensive routines were written in C. If you prayed to the correct gods and uttered the necessary incantations, you could use Microsoft's CodeView debugger to debug the C code. But usually not for more than a few minutes, as the program usually didn't like to spend much time running with CodeView (usually memory problems).
I had a series of makefile switches that I used to stub out the sections of code that I didn't need to debug at the time. My debugging environment was very sparse so there was as much free memory as possible for the program. I also think I drank a lot more back then...
Some years ago I reverse engineered game copy protections. Because the protections was written in C or C++ they were fairly easy to disassemble and understand what was going on. But in some cases it got hairy when the copy protection took a detour into the kernel to obfuscate what was happening. A few of them also started to use of custom made virtual machines to make the problem less understandable. I spent hours writing hooks and debuggers to be able to trace into them. The environment really offered a competetive and innovative mind. I had everything at my disposal save time. Misstakes caused reboots and very little feedback what went wrong. I realized thinking before acting is often a better solution.
Today I dispise debuggers. If the problem is in code visible to me I find it easiest to use verbose logging. (Sometimes the error is not understanding the interface/environment then debuggers are good.) I have also realized time is of an essance. You need to have a good working environment with possibility to test your code instantly. If you compiler takes 15 sec, your environment takes 20 sec to update or your caches takes 5 minutes to clear find another way to test your code. Progress keeps me motivated and without a good working environment I get bored, angry and frustrated.
The last job I had I was a Sitecore Developer. Bugfixing can be very painful if the bug only occurs on the client's system, and they do not have Visual Studio installed on the system, with the remote debugging off, and the problem only happens on the production server (not the staging server).
The worst in recent memory was developing SSRS reports using Dundas controls. We were doing quite a bit with the grids which required coding. The pain was the bugginess of the controls, and the lack of debugging support.
I never got around the limitations, but just worked through them.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Did you ever had a bug in your code, you could not resolve? I hope I'm not the only one out there, who made this experience ...
There exist some classes of bugs, that are very hard to track down:
timing-related bugs (that occur during inter-process-communication for example)
memory-related bugs (most of you know appropriate examples, I guess !!!)
event-related bugs (hard to debug, because every break point you run into makes your IDE the target for mouse release/focus events ...)
OS-dependent bugs
hardware dependent bugs (occurs on
release machine, but not on
developer machine)
...
To be honest, from time to time I fail to fix such a bug on my own ... After debugging for hours (or sometimes even days) I feel very demoralized.
What do you do in this situation (apart from asking others for help which is not always possible)?
Do you
use pencil and paper instead of a debugger
face for another thing and return to
this bug later
...
Please let me know!
Some things that help:
1) Take a break, approach the bug from a different angle.
2) Get more aggressive with tracing and logging.
3) Have another pair of eyes look at it.
4) A usual last resort is to figure out a way to make the bug irrelevant by changing the fundamental conditions in which it occurs
5) Smash and break things. (Stress relief only!)
I once worked for a company that sold a client-server application that was basically a file transfer and synchronization tool. Both the client and the server were custom applications we had designed.
We had a persistent bug that was very hard to duplicate in the lab. Our server could only handle a certain number of incoming client connections per box, so many of our customers would "cluster" multiple servers together to handle large user populations. The back end data for the cluster was on a file server they all shared. In this cluster configuration there was a bug that would happen under load where we would get a low-level file system error code on a file sharing call involving one of the back end files. Nobody could get this to repeat reliably in the lab, and even when they could they couldn't narrow down what was happening.
(I forget the exact error, it was probably 59 ERROR_UNEXP_NET_ERR or maybe 65 ERROR_NETWORK_ACCESS_DENIED. As I recall it was not even one of the documented error codes you were supposed to be able to get from the API we were calling, which was usually a lock or unlock call on a file section).
Since it involved the communication between the server and the back-end file store, and I was the "network transport" guy, I was tasked with looking at it. Many others had looked at it with no luck.
The one solid thing I had was I knew where in the code the error was being detected, but not what to do about it. So I needed to find the root cause. So I set up an appropriate hardware environment to duplicate it, and I put a custom build of the server software that instrumented the section of code in question.
The instrumentation was as follows: I added a test for the troublesome error code, and had it call a piece of code to send a UDP packet to a predetermined network address when the error occurred. The UDP packet contained a unique string in it to key on.
I then set a packet sniffing tool on the network. (At the time I was using Microsoft Network Monitor). I positioned it where it would be able to "see" this UDP packet when it was sent as well as all the communication between the cluster servers and the file server.
Most good sniffers have a mode where you can have it capture until it sees a particular piece of traffic, then stop. I turned on that mode and set it to look for that UDP packet my code would send. The goal was to end up with a packet capture of all the file server traffic right before the bug occurred. The very last network packets to and from the system where the UDP packet originated would presumably be a big clue as to what was happening.
I set the "stress test" configuration going and went home for the weekend.
When I got back on Monday, lo and behold I had my data. The sniffer had stopped just as expected after many hours of running and contained a capture. After studying the capture, what I found was that the Server Message Block or SMB (aka CIFS aka SAMBA) connection between our server and the file server was actually timing out at the TCP level due to extreme loading on the server. Because all of Microsoft's stuff is heavily layered, it would percolate back up through the file sharing stack as an "unexpected" error instead of returning a more intelligible error code that said "hey, you lost your connection at the TCP level".
I did a little more research on the TCP settings for Windows, and lo and behold the defaults for the version of Windows we were using (probably NT 4 in that era) were none too generous. It was only allowing for a very small number of failures on the TCP connection and boom, you were dead. Once you lost your SMB connection to the file server, all your file locks were toast and there was no way to recover.
So I ended up writing an appendix to the user manual that explained how to alter the TCP settings in Windows to make your cluster server a bit more tolerant of high load situations. And that was it. The fix to the bug was zero change in code, merely some additional documentation on how to properly configure the OS for use by this product.
What have we learned?
Be prepared to run altered versions of your code to investigate the problem
Consider using non-traditional tools to solve the problem (sniffers)
Not all bug fixes require code changes
Sometimes you can diagnose a bug while at home having a beer
I do a number of different things:
throw out all my assumptions and start from scratch. Remember, a bug exists because something which appears correct is actually wrong. Even those lines or functions or classes that you are absolutely certain are correct may be incorrect. Until you can convince yourself of the correctness you can't assume anything is right.
keep putting in print statements and assert statements to eliminate things and allow me to reform new assumptions.
step through code in the debugger if the problem is a control flow problem. Don't step over functions. Step in them and go through all the detail of their execution to confirm they are working right. Confirm the arguments and return values.
If a line or function or class is suspect but I can't prove it in situ, then write a small test case that does what you think the problem construct does. This may locate the problem or give some insights as to where to look next.
stop for the day. It's amazing what kind of offline processing your brain will do overnight. Often the answer or a key insight will appear the next day while I'm doing something mindless like showering or driving.
Create an automated way to cause the bug. The worst bug to fix is one that takes hours to reproduce.
Quote taken from "The Cryptonomicon":
"Intuition, like a flash of lightning, lasts only for a second. It generally comes when one is tormented by a difficult decipherment and when one reviews in his mind the fruitless experiments already tried. Suddenly the light breaks through and one finds after a few minutes what previous days of labor were unable to reveal."
I usually ask someone else to take a look at the code. While I'm explaining what the code is supposed to do, I sometimes see the bug just as I talk.
When a bug is a tough one, I sit and work until I figure it out and solve the problem. Interestingly enough, there are times when catching a mysterious bug is more enjoyable than everything running smoothly. And the relief and feeling when a bug is resolved, well, not many other things can beat that (except the obvious ones).
If all else fails, don't tackle it directly. Rewrite the problem area code in a more refactored way.
I have definitely had bugs which I worked on for 4-5 days continuously before finding a solution. Other bugs have sat in the bug tracker for months, as I put in a few hours spread out over a long period of time. I think this sort of bug is inevitable in any complex software project.
Some stuff that works well for me:
binary search through the program flow with logging
use Trace statements along with DbgView to search for bugs which show up in release mode
find an alternate way to reproduce the bug without changing the code
(works against logic, but...) change the code so that the bug is more easily reproducible (the failure condition is more readily achieved)
sleep on it and try again tomorrow with a fresh pair of eyes :)
The worst sort of bug in my opinion is a concurrency bug which disappears when logging is inserted.
Lots of great answers here. One thing that's worked for me in the past is to ask "what can I do to make it totally obvious when this problem has occured?".
For example, if the problem is a corrupted value in a data structure, try building a consistency-check routine that you can run periodically. Also consider implementing all access to the shared data through a set of functions that log each change.
Or, if the problem is a "random" memory overwrite, use a replacement malloc()/free() implementation that traps writing to "free" memory (like electric fence or dmalloc).
Someone else mentioned automating the process of triggering the bug. This is greeat if you can do it. Even having a routine that randomly exercises the program might help in these cases.
Seriously? I do things in this order.
Go to bed
Ask a colleague
Rewrite so the area isn't affected.
Ask SO
Raise a support ticket with your 3rd party library vendor.
"What do you do in this situation (apart from asking others for help which is not always possible)?"
When is it not possible to ask for help?
There are always others you can turn to for assistance - your coworkers, your boss, friends here at Stack Overflow, etc.
Understanding when to seek help shouldn't be demoralizing!
There are a lot of good tips here.
One that I absolutely do not agree with is the concept of changing the code hoping that it will go away. First off, you a probably going to introduce new bugs. Seconds, you can easily change things enough to hide the bug only to have it resurface again with the next patch.
Memory corruption bugs are especially likely to vanish as magically as they turn up. However, the memory corruption bug isn't fix, it is only that non-fatal areas of memory are getting trashed.
1) Try a different debugger. For example, I use WinDbg more and more often. When you load a program in a debugger, memory layout for your application will change slightly. Maybe a different debugger cause the error to manifest slightly differently.
2) If you resort to changing code without knowing exactly what the problem is, then if the bug goes away, YOU MUST go back and understand why the change fixed the bug. Otherwise, you are probably just hiding the bug.
3) Talk to others about the bug, maybe they have seen different versions of the same problem (i.e. other ways to recreate it)
4) Logging.
I've had bugs that took weeks or months before a solution was found, but eventually all bugs do get fixed. Aside from the classical non-debugger bug-tracking techniques like disabling parts of the system until you get a minimal test case, I've used these techniques:
Looking for better debugging tools. A new perspective goes a long way. Xdebug is something I started using in PHP only because of a performance bug that I wasn't making headway on.
Studying the technology that the bug is located in. This has helped to debug an outlook add-in. It had random errors that made no sense and that google searches turned up zilch about. By researching outlook add-in best practices, COM and MAPI programming, we got a clearer picture of what could go wrong, and thought of new things to try to fix the bugs, which eventually did fix them.
Trying to exacerbate the problem. If there's an issue that only happens occasionally, I'll try to find ways to make it happen constantly. This has helped to track down errors in web apps under IE and also to narrow down a crashing bug in the flash plugin.
When all else failed, I've rewritten the subsystem that caused problems from scratch. This may take a few days, or even weeks, but if you're stuck on a bug, and can't resolve it, and customers won't take no for an answer, what else can you do? This doesn't always fix things, but if it doesn't, you usually get a clearer picture of what's going wrong.
I've noticed a few commonalities in these bugs that I get stuck on for weeks:
Asking 3rd parties for help rarely helps, and it's generally not a good idea to wait for someone else to come save the day.
Almost always the fault is inside some 3rd party closed source technology, especially when using obscure parts. IE had nasty bugs when trying to use client certificates. Flash didn't deal well with randomly generated drawing instructions (some of which were nonsensical). Outlook doesn't like it when you try to change form layout dynamically from code. These days I've learned to respect the "comfort zones" of proprietary tech.
I give it more time. I once had a bug (in a personal project) that I just could not figure out. I tried every debugging method I could think of, including Google, with no success. Six months later, I came back and found the bug within an hour or so. It wasn't something simple (something apparently undocumented was going on deep inside Swing), but I just looked at it in a way I hadn't before.
I've had this problem before, I believe everyone has, I have flat out given up before, it was simply impossible to find, yet it kept crashing, when theres some kind of bug in the code, what I do is just sit down and concentrate on every bit of the code little by little until I find it, it's hard and it takes patience but it's all you can do in such a situation.
Hope this helps.
I honestly cannot recall a bug that I couldn't fix. It may cause a lot of refactoring, or may take a while, but I've never had one that I can't get rid of. If it takes me more than an hour to track it down then it's almost always something really stupid and small like looking right past that : that should've been a ;, etc.
In python, if I'm using an editor that isn't mine, or maybe it's someone else's code, I use retab! in vim, or paste into something like pastie to check indentation (if I don't have vim available).
If it's not a crasher/deal breaker, then I move on and come back with a fresh pair of eyes.
Oh, and you can never, ever have too much logging.
I add as much debug as possible (write to log file, message boxes, etc.), and test.
I don't think this is the worst bug you can find. The worst ones are those you can't reproduce deterministically or in the testing environment.
I get a bit demoralized too when unable to solve a bug. Usually when I hit a wall with a bug, I would just take note on my findings and stop working on it. I would jump on another bug that is easier to solve and then came back to the bug. By doing this, I would have a fresh mind and attitude in tackling the bug. Sometimes you might have tendency to overcomplicate things when spending too much times on a bug. Having a break, helps in breaking the wall.
RWendi
First off, is it reproducible? That's a HUGE plus if it is. I want things bugs to always/never happen... its the intermittent ones that are the troublesome ones.
And it is going to depend on the problem, but at my shop we'll generally tag-team such a problem figuring that 2 heads (or 3 or 4) is better than 1.
Occasionally the bug won't even be in MY code, but it generally is. There have been issues where a 3rd party library was the culprit or a particular implementation on a particular platform was the cause - those stink.
I'll use anything and everything to at least track it down: debuggers, trace output, whatever.
Typically, if I can isolate it to a class or module I'll write a test harness to duplicate the real world and try to duplicate it there. I generally write my test code first, but sometimes legacy code (or other developer's code) exists that doesn't have tests already.
I generally will talk the design and problem through, out loud with the team and whiteboard anything that isn't clear. Often the solution will bubble to the surface once we talk about it as a group.
That's what I do.
I usually, try hard solving it. But, if that is not possible for reasonable windows of time, I leave it for some time to braincells to solve it while i sleep ;) Sometime it works...
I've considered asking for help on this website called StackOverflow that I've been frequenting lately...
This is what I did today...
I debug HW/SW interaction and its often the case logging (instrumentation) changes or hides the bug. Hence tests are performed "at-speed". I call these bugs "roaches" as they run away from any light I can shine on them.
So I have to:
Find the transaction that causes the bug. List the HW interaction via logging (this test passes, but it illustrates the flow).
Instrument before and after the bug to print state changes.
The bug I'm solving now of course is worst case as the HW locks up. The HW includes the CPU so its like being in a well lit room then the power fails and its pitch black.
I have a special backdoor view into memory, but of course this is locked up also. I tried power cycling in the hopes that the memory would stay non-volatile long enough to reenable the backdoor. No such luck. This is possible though.
I very very carefully wrote all the steps I went through to characterize this bug (what works, what fails etc). Sent this to developers with similar HW to verify it just wasn't me or my HW.
I took a few hours break to let this info settle and see if any lightbulbs lit elsewhere.
No replies, this bug is mine to solve...
This HW SW interaction is a loop tha does some setup then enters a polling loop that reads when the transaction is finished. Many transactions should occur. Which transaction fails? Is it the first one (indicating I can debug the transaction and not some noise in the HW). Is it the always the Nth transaction? What makes the Nth different than the first or the (N-1)th. The SW is single threaded and built to be predictable. No preemption, no interrupts enabled.
This SW has worked before, whats new? All the HW is new. In this case all the silicon is new as its an ASIC. Even the embedded CPU is new and customized so the ISA is new.
So I suspect everything and I'm blind. I'll have to sneak up on this roach.
I enabled just the log that reports how many times the SW polls the HW for completion. In this way the first transaction runs at speed, I get an idea how often I touch the HW in a tight polling loop. The test passes. I know its the Nth transaction and I recorded the peak number of polls for all transactions (perhaps meaningless data).
After modifing anythin, I have to put it back the way it was to verify the bug still exists. After all the earth has rotated and the solar winds are not as strong ;)
Looked at all the checkins, saw a contractor changed some important setup parameters with no explanation. These (outsourced) people are still under evaluation. This will not help.
Found there was no spinwait in the polling loop. Bad for the loop timeout as without it the timeout depends on CPU speed. Added spinwait, still no happiness.
Limited the number of transactions to see where it fails, somewhere before 1000.
Setup the HW to run slower, still hangs.
Hate to leave anyone reading this hanging too, but this diatribe will have to wait till tomorrow.
There is no bug that can't be fixed, since there is no bug that can't be fixed with a total rewrite.
An unfixable bug is just a bug you aren't willing to replace.
For memory related bugs i have found that the Memory Profiling options of Ants Profiler have helped me quite a bit on finding bugs.
use more creative methods of tracking the bug down.
using remote debugging on the machine where its reproducable.
using profiling tools.
introduce more logging to the app.
Going away for a while and then coming back to a problem is one common approach I do and have heard.
How easily reproduced the bug is can be a factor as well since if the error only occurs in one in a zillion runs of a program that could be considered a negligible gain for fixing it by breaking something else.
There is also the question of nailing down where the bug is, is it in some configuration so that it occurs on a server but not my local XP Pro machine which runs IIS 5.0. Some other bugs may involve having to change the resolution of my machine that can be annoying to try to reproduce a bug that others have reported.
You left out the "occurs under another O/S" category of bugs so that a web page that is fine in IE and Firefox on PC may look like crap on Safari on a Mac. Do I get my hands dirty in trying to fix a CSS issue using my machine as a server and the Mac that is over a row or two in the cubicles of the floor in order to see this issue or is it so low a priority it gets swept under the rug? Alternatively, if a bug was on Linux and there aren't any Linux machines near me, what should I do?
I'm sorry to have left with some questions but these seem to be difficult questions for me at times.
In addition to the debugger, I've also used logging and old fashioned paper and pencil. On occasion I've found really hard bugs, like code that runs fine in debug mode, but breaks in release mode. I've even occasionally rewritten perfectly good code that for whatever reason, doesn't work reliably, figuring that it's better to be reliable than elegant.
I sometimes try to redefine what others term a bug as really being a feature, but that seldom works!
I have a bug that shows up every few months on a customer site. It usually happens at 3am and it's not discovered until early the next morning when the customer arrives at their site. And usually when they discover it, they want everything to get working immediately, so our support people generally just reboot the computer. It's been driving me nuts for years. It never happens on my test machine or in the QA lab, only at certain customer sites. Over time, I've
refactored some of the code that I thought was causing it
added more debugging printouts around where it appears to be crashing
redirected stdout so that next time I see it I can "kill -3" the process
given support some new tools to dump out the current state of database locks and the like.
added diagnostics to make it more obvious when it does happen
It hasn't happened in a few months, and I've got my fingers crossed that I might have fixed it this time, but I'm not counting on it.
If it's not critical, don't fix it, you'll just spend too much time!
Keep the bug open. comment/work on it when you can. It might get fix by accident (or by someone else) later on!
Sometimes it takes a little lateral thinking, but every bug is fixable. Sometimes you need to leave it and sleep over it, sometimes it's good to ask someone else to have a quick look (they may see something you haven't), but mostly it's about trying different things, calling up on previous experience. It can be frustrating, but the buzz you get when you do fix it, is like no other!
I have a crash dump of an application that is supposedly leaking GDI. The app is running on XP and I have no problems loading it into WinDbg to look at it. Previously we have use the Gdikdx.dll extension to look at Gdi information but this extension is not supported on XP or Vista.
Does anyone have any pointers for finding GDI object usage in WinDbg.
Alternatively, I do have access to the failing program (and its stress testing suite) so I can reproduce on a running system if you know of any 'live' debugging tools for XP and Vista (or Windows 2000 though this is not our target).
I've spent the last week working on a GDI leak finder tool. We also perform regular stress testing and it never lasted longer than a day's worth w/o stopping due to user/gdi object handle overconsumption.
My attempts have been pretty successful as far as I can tell. Of course, I spent some time beforehand looking for an alternative and quicker solution. It is worth mentioning, I had some previous semi-lucky experience with the GDILeaks tool from msdn article mentioned above. Not to mention that i had to solve a few problems prior to putting it to work and this time it just didn't give me what and how i wanted it. The downside of their approach is the heavyweight debugger interface (it slows down the researched target by orders of magnitude which I found unacceptable). Another downside is that it did not work all the time - on some runs I simply could not get it to report/compute anything! Its complexity (judging by the amount of code) was another scare-away factor. I'm not a big fan of GUIs, as it is my belief that I'm more productive with no windows at all ;o). I also found it hard to make it find and use my symbols.
One more tool I used before setting on to write my own, was the leakbrowser.
Anyways, I finally settled on an iterative approach to achieve following goals:
minor performance penalties
implementation simplicity
non-invasiveness (used for multiple products)
relying on as much available as possible
I used detours (non-commercial use) for core functionality (it is an injectible DLL). Put Javascript to use for automatic code generation (15K script to gen 100K source code - no way I code this manually and no C preprocessor involved!) plus a windbg extension for data analysis and snapshot/diff support.
To tell the long story short - after I was finished, it was a matter of a few hours to collect information during another stress test and another hour to analyze and fix the leaks.
I'll be more than happy to share my findings.
P.S. some time did I spend on trying to improve on the previous work. My intention was minimizing false positives (I've seen just about too many of those while developing), so it will also check for allocation/release consistency as well as avoid taking into account allocations that are never leaked.
Edit: Find the tool here
There was a MSDN Magazine article from several years ago that talked about GDI leaks. This points to several different places with good information.
In WinDbg, you may also try the !poolused command for some information.
Finding resource leaks in from a crash dump (post-mortem) can be difficult -- if it was always the same place, using the same variable that leaks the memory, and you're lucky, you could see the last place that it will be leaked, etc. It would probably be much easier with a live program running under the debugger.
You can also try using Microsoft Detours, but the license doesn't always work out. It's also a bit more invasive and advanced.
I have created a Windbg script for that. Look at the answer of
Command to get GDI handle count from a crash dump
To track the allocation stack you could set a ba (Break on Access) breakpoint past the last allocated GDICell object to break just at the point when another GDI allocation happens. That could be a bit complex because the address changes but it could be enough to find pretty much any leak.