Chapter 6. Fun with Crash Dumps

Music for Debugging

Bugs Never Disappear

If we complain about heap corruption again or type !locks every day and see pages of output filled with columns the following album from Supertramp reminds us that

Some Things Never Change

Here is my track name interpretation:

  1. It's a Buggy World

  2. You Code, I Debug

  3. Let's Debug Together

  4. Live to Debug It

  5. Some Bugs Never Disappear

  6. Read My Bug Report Please

  7. Sooner or Later I Fix It

  8. Help Me Down that Code Path

  9. And the Customer

  10. Il Est De Mon Bug!

  11. Where There's a Bug

Horrors of Computation

The author was a big fan of Kitaro music for many years during 90s. After re-listening to some of his albums he recommends "Heaven & Earth: Music From The Motion Picture Soundtrack"to listen to when analyzing complete memory dumps from complex issues, looking at user (Earth) and kernel (Heaven) spaces and trying to figure out which module started the conflict.

Passion, Intellect and Expression

If you like the author totally immerse in complete memories of computation you probably would like complete works of Beethoven.

Headphones for Debugging

Some fellow debuggers sometimes ask the author what brand of headphones he uses during debugging and memory dump analysis sessions. It depends on the working environment. In the office, at the time of this writing, the author uses STAX[23] electro-static headphones (sounds like STACKS!), one of the previous versions of their Basic System, similar to this one:

http://www.stax.co.jp/Export/SRS2050II.html

The author learnt about STAX in 2000, Moscow, when he was obsessed with pure sound and rushed to the nearest dealer to buy the old version of SR-001:

http://www.stax.co.jp/Export/SR001mk2e.html

In the Memory Dump File

The author used to be a fun of Yanni[24] music in the late 1990s. He started listening again to some of his albums and now recommends them for any debugging session. If you are new to the music of Yanni there is a compilation album: In the Mirror.

Here is the author version of track titles inspired by listening (with my comments in italics):

  1. In the Memory Dump File

  2. The Morning Session

  3. Love for Debugging

  4. A Debugger's Dream

  5. Within Kernel

  6. Forbidden Access

  7. Once Upon a Second Chance

  8. Chasing Bugs

  9. The Main Thread [Special Debugging Version]

  10. Quiet Memory Analyst

  11. Debugging Joy (My Life is Debugging)

  12. So Long My Debugger (My Only Friend on Virtual Memory Plains)

  13. Before I Leave the Debugger

  14. End of Session (It wasn't bad after all)

  15. Face in the Memory Dump (after applying Natural Memory Visualization techniques: you can see pictures and various artifacts stored in memory buffers, Volume 1, page 532)

Bugteriology

The study of crash dump analysis patterns needs its own name and the obvious choice is Bugteriology:

Bugteriology is the study of crash dump analysis patterns (bugteria). Its main subject is identification, classification and characterization of such patterns found in memory dumps (bugterial species).

During one of those days when the author was in a good mood thinking about bugs, suddenly a thought stroke him about the similar sounding words bacterium and bugterium (perhaps because he was reading a theoretical biology book, Essays on Life Itself). We admit that it might be sounding the same only for a non-native English ear though. So the new definition was born:

Bugterium (pl. bugteria) - an instance of a memory dump analysis pattern found in a crash (memory, core) dump file.

Why a bugterium and not a cdarium? The motivation (with hindsight) lies in the complexity of debugging (and life forms). While a bug is a complex thing (and a beast) and it takes sometimes days or weeks to chase and fix (kill) the one, a bugterium (bacterium) is of relatively smaller complexity and can be easily identified and dealt with by component removal or upgrade (massively killed). From software technical support perspective, remember this bugtation No.14:

Crash dump analysis "is anticipated with" joy "performed with" eagerness, "and bragged about forever."

Although the perceived simplicity of crash dump analysis is deceptive (bugtation No.2):

"It requires a very unusual mind to undertake the analysis of the obvious" crash.

Alfred North Whitehead, Science and the Modern World

Implausible Debugging Book Titles

I found the book "How to Avoid Huge Ships: And Other Implausibly Titled Books" (ISBN-13: 978-1-84513-321-4) in a local bookshop and couldn't stop laughing. So I took some implausible titles and bugtated them into implausible debugging book titles:

  • Old Bugs and the Men Who Debug

  • Them How to Avoid Crashes and Hangs or I've Never Met a Bug I Liked

  • Blue Screen: What's in it for You?

  • What to Say When You Debug: Powerful New Techniques to Program your Success!

  • Redmond: The View from Greenland

  • Fabulous Small Bugs

  • Better Never to Have Coded: The Harm of Coding

  • Code for Impact

  • Whose Bug? The Clash between Software Vendors

Build Date Astrology

I often hear about cosmic mysteries or influences when problems happen in computer environments. Passing by an astrology section in a local book shop a revelation came to me that a compile / link time (build time) might influence a component (DLL, EXE, SYS files), product or system behaviour. I'm also going to publish a book for this iterative and incremental activity called:

Title: The Secret Language of Build Dates: Unique Astrology Profiles for Every Build of the Year with Advice on Testing, Troubleshooting and Debugging ISBN-13: 978-1906717407

Knowing build dates will help you to test, troubleshoot and even debug software in hopeless cases where you don't know where to start. Astrology will help you to choose a random direction! Finally the output of WinDbg lmv command has more sense to me Build Date Astrology

Breaking Technical Barrier

Sometimes there are difficulties to explain the nature of software faults to not so technically oriented customers or casual inquirers. Here the language of software astrology (www.SoftwareAstrology.com) can greatly help in assembling the right phrases and provide easily understandable analogies.

Occult Debugging

Once upon a time I was analyzing a memory dump and in the process become upset. I realized that the dump was vacuous and afterwards learnt that a customer forced the dump of a normal system with steady-state computational activities inside. At the same time computational precognition, the ability to see in advance what would happen with the system, always fascinated me from the very beginning of my memory dump analysis work. In computational precognition phrase, the word "computational" refers to computers and "precognition" refers to humans but this should not be the problem as the boundary between these two categories is not sharp and can be shifted in either direction. Investigation of occult, paranormal and supernatural in the realm of computable is the domain of psi-computation.

The Year of Dump Analysis!

DumpAnalysis.org portal announced 2010 as The Year of Dump Analysis. Here is the short Q&A session portal founder.

Q. Why 2010?

A. Two reasons: 1) To do dump analysis effectively and efficiently an engineer needs some experience in debugging acquired in the previous year of debugging (perhaps after 7 debugging nights, 2009, 0×7D9); 2) 2010 is 0×7DA.

Q. What is the meaning of 7?

A. It is interpreted as Dump Analysis 7 days a week. Like what I do. Or from kernel pool tag perspective it is AD7: Analysis of Dumps 7 days a week.

Q. What about the year 2011, 2012, 2013? 0×7DB, 0×7DC and 0×7DD?

A. Hmm, sounds like WinDbg commands db, dc and dd...

Stack Traces and Poetry

We can read stack traces like English verse (from bottom to top):

0:01> ~8kL
ChildEBP RetAddr
009ef258 7c827d0b ntdll!KiFastSystemCallRet
009ef25c 7c83d236 ntdll!NtWaitForSingleObject+0xc
009ef298 7c83d281 ntdll!RtlpWaitOnCriticalSection+0x1a3
009ef2b8 7c82dabf ntdll!RtlEnterCriticalSection+0xa8
009ef358 7c82dab1 ntdll!LdrpGetProcedureAddress+0x128
009ef374 77e764ea ntdll!LdrGetProcedureAddress+0x18
009ef5d8 7c34c456 kernel32!UnhandledExceptionFilter+0x46f
009ef5f4 7c34957c msvcr71!_XcptFilter+0x15f
009ef600 7c34246e msvcr71!_endthreadex+0xb7
009ef628 7c828752 msvcr71!_except_handler3+0x61
009ef64c 7c828723 ntdll!ExecuteHandler2+0x26
009ef6f4 7c82855e ntdll!ExecuteHandler+0x24
009ef6f4 7c82be3e ntdll!KiUserExceptionDispatcher+0xe
009efa00 7c82a319 ntdll!RtlpFindEntry+0x68
009efc2c 7c3416b3 ntdll!RtlAllocateHeap+0x606
009efc6c 7c3416db msvcr71!_heap_alloc+0xe0
009efc74 7c360947 msvcr71!_nh_malloc+0x10
009efc80 0285f893 msvcr71!operator new+0xb
009efca8 02852e38 SQLModule!ODBCDelete+0xf3
009efd54 0269acff Store!ProcessDeletes+0x3d
009eff38 0269badb Store!UpdateStore+0xe
009eff58 00323499 Common!WorkItem+0x15c
009eff84 7c349565 Common!WorkItemThread+0x339
009effb8 77e64829 msvcr71!_endthreadex+0xa0
009effec 00000000 kernel32!BaseThreadStart+0x34
  • The new thread started

  • To work through items

  • It got an item

  • Handled it to store

  • To run delete requests

  • Through Oh-Dee-Bee-See

  • It tried to alloc

  • But crashed in malloc

  • While browsing the heap

  • Exception was dispatched

  • And handler called at once

  • But couldn't find a filter

  • And called default one

  • That filter needed help

  • And looked for its address

  • But halted in suspense

  • While entering crit sec.

Debugging Slang

a bloke - a blown kernel

Examples: I see a bloke. I got this company's bloke.

American equivalent of a British bloke:

a pal - a panic alarm

Examples:I see a pal. I got this company's pal.

Memory Dump Analysis Walks

One day, the author was walking in Malahide[25] Woods and thinking through his dangerous ideas about universal memory dumps and how to reconcile man-made PDB files with empirically discovered science files (page 301). Upon finding a problem resolution, the author sat firmly on the ground and remained there happily for some time.

Memory Dump Analysis Walks
Memory Dump Analysis Walks

E-Acheri

This is a computational ghost that comes from CPU spiking tops and hills to bring sick-ness to small programs. Red color from RGB triplets provides early warning signs. This was inspired by non-computational Acheri[26]. Computational ghosts and bug hauntings were inspired by an e-mail conversation between two software engineers that I witnessed some years ago when one told another: "Is this your fix that is still haunting us?" If you have ever visited heated code review debates you can imagine the provoked response.

The Meaning of DATA

Unification of software traces (page 308) with memory dumps (Volume 1, page 501) suggests that DATA is simply Dump Analysis + Trace Analysis. It is commutative with TADA, the sound of accomplishment (tada.wav in Windows Media folder).

This can't be a coincidence, can it?

Irish Government on Dumps

Whom to blame if a process or a system crashes or freezes?

"Mr. Quinn: It is very dangerous to ignore political dumps these days. Mr. Dempsey: All dumps are political."

Source[27]

Memory Dumps as Relics

Some memory dumps can serve the role of a relic[28] and be the subject of veneration. For example, a universal memory dump (page 295) that reveals the eternity and infinity of Memory. Speaking about earthly artefacts, like computer memory dumps, some of them could be relics and subjects of personal veneration after being generated from a cult system or when a venerated person was working on a computer, for example, doing code construction or writing and composing great works of significant value.

The Ghost of Adelphi Training Center

The Adelphi Training Center in Dublin, Ireland, is haunted by the ghost of pre-Internet epoch. During one debugging night, when installing a secret service, Drilliam Traceless was murdered by a redundant engineer who was envy to his charisma. He stubbed him in the back while Drilliam was unlocking his mini-computer case. He died whispering: "I'll be back debugging". His face sometimes appears on greenish screensavers running on computers located in that center. During morning training sessions, many trainees think this is a kind of the so called Guinness effect.

The story is adapted from The Ghost of Adelphi Theatre[29].



[23] http://www.stax.co.jp/

[24] http://en.wikipedia.org/wiki/Yanni

[25] http://en.wikipedia.org/wiki/Malahide

[26] http://en.wikipedia.org/wiki/Acheri

[27] http://historical-debates.oireachtas.ie/D/0488/D.0488.199803040007.html

[28] http://en.wikipedia.org/wiki/Relic

[29] http://en.wikipedia.org/wiki/Adelphi_Theatre

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset