This page is useful for advanced debugging tricks that we've found over the years, each hint has a description of the problem we were trying to face, and how we solved it. If these tools are way over your head, try looking at the DeBugging page instead, it covers a lot of the more simple tools.
We once had a bug in ircu where it would crash in one piece of code, something had obviously been corrupted, but this code was obviously not the cause of the bug, but a value was being set incorrectly just before hand.
Kev's solution was to write a program called "ringlog". ringlog links to your program and uses the profiling hooks so that it gets called on every function entry and exit. It then writes the function into a circular ring buffer, which is saved to disk. When the program crashes then you can find out the recent history of the program (what functions were called) to find the bug. Kinda like a traceback on steroids.
The tool is available under tools/ in the ircu cvs repository: 
http://cvs.undernet.org/viewcvs.py/undernet-ircu/ircu2.10/tools/
This is a variation of the above. We had a variable that was being trashed in a program and didn't know where. In fact, we couldn't see where anything should be modifying the variable. Using gdb to put a test on write of a memory location caused gdb to make the program slower than your average iceage. Our solution was to use the same hooks and compare the variable every time the function entered or exited, and raise an assertion if it ever changed. Another variation on this technique is to write a function to validate a list, and raise an assertion of the invariant is violated. This doesn't get you the exact line of code that the error occurs on, but it does get you to the nearest function.
The tool is available http://coders.meta.net.nz/perry/tools/invariant.tar.gz
valgrind lets you set r/w permissions per byte. In the above issues you can flag a byte as being r/o and let valgrind raise an assertion when that memory is written.
We had a program that was running fine one day, then afew days later was being dog slow. We wanted to quickly and easily figure out why the program was so slow. By attaching gdb to it, stopping it at random, and looking at where the program was, and then continuing it. If you do this about 10x, you'll get a feel for what functions it's sitting in. The probability of stopping it anywhere says that you're more likely to stop it in a function that's taking most of the CPU.
Ever wanted to know what the state of a program is, without having to stop it? This is particularly important for long running programs. Add some code (or use gdb) to make it fork() and have the child dump core. The parent can even wait for the child and rename the core file based on the time, or what it was doing. This is a handy way for having an "almost" assertion. An assertion that fires, but doesn't kill the program (eg, because you know how to recover from it, but it shouldn't be occuring).
Beware that dumping core is often a slow and intensive process and may impact performance of your system, particularly disk-poor systems.
This runs some theoretical program with file.input as an argument, breaks in do_stuff, displays g_my_stuff every time the program stops, continues twice past the do_stuff breakpoint, sets a temporary breakpoint in set_stuff, continues till it hits this breakpoint, then continues until line 192. All of this is automated when you type gdb programname in the directory the .gdbinit resides in.
Sometimes you hypothesize that the compiler is generating broken assembly (or generating assembly you didn't expect)
objdump -l -S --disassemble -r ./foo
Will display the assembly intermixed with the source (if it was compiled with debugging symbols).
2 pages link to AdvancedDebuggingHints:
