This page is useful for advanced debugging tricks that we've found over the years, each hint has a description of the problem we were trying to face, and how we solved it.
We once had a bug in ircu where it would crash in one piece of code, something had obviously been corrupted, but this code was obviously not the cause of the bug, but a value was being set incorrectly just before hand.
Kev's solution was to write a program called "ringlog". ringlog links to your program and uses the profiling hooks so that it gets called on every function entry and exit. It then writes the function into a circular ring buffer, which is saved to disk. When the program crashes then you can find out the recent history of the program (what functions were called) to find the bug. Kinda like a traceback on steroids.
The tool is available under tools/ in the ircu cvs repository: 
http://cvs.undernet.org/viewcvs.py/undernet-ircu/ircu2.10/tools/
This is a variation of the above. We had a variable that was being trashed in a program and didn't know where. In fact, we couldn't see where anything should be modifying the variable. Using gdb to put a test on write of a memory location caused gdb to make the program slower than your average iceage. Our solution was to use the same hooks and compare the variable every time the function entered or exited, and raise an assertion if it ever changed. Another variation on this technique is to write a function to validate a list, and raise an assertion of the invariant is violated. This doesn't get you the exact line of code that the error occurs on, but it does get you to the nearest function.
The tool is available http://coders.meta.net.nz/perry/tools/invariant.tar.gz
valgrind lets you set r/w permissions per byte. In the above issues you can flag a byte as being r/o and let valgrind raise an assertion when that memory is written.
2 pages link to AdvancedDebuggingHints:
