PART 6: Software Diagnostics, Root Cause Analysis, Troubleshooting, and Debugging

Diagnostics of Things (DoT)

We introduced Narratology of Things as a combination of Software Narratology of Things (Introducing Software Narratology of Things (Software NT), Volume 7) and Hardware Narratology (Unified Computer Diagnostics: Incorporating Hardware Narratology, Volume 7). Since memory dump analysis may be considered as a part of general trace and log analysis (Special and General Trace and Log Analysis, Volume 8b) we open a new research direction for Diagnostics of Things (DoT) based on Narratology of Things and Pattern-Oriented Trace and Log Analysis, which also includes software execution artifacts of things and pattern-oriented network trace analysis (Pattern-Oriented Network Trace Analysis, ISBN: 978-1908043580) from IoT.

Riemann Root Cause Analysis Language

Image generated by 3D-XplorMath

Incepted and named in February 2009 shortly before the first software trace and log analysis pattern (Volume 3) was published in April the same year, Riemann Programming Language (Volume 3) was thought of as a software problem description language (Volume 7) capable of generating software problem-solving tools (including TaaS version, Volume 7). A book was planned for publication in 2010 (The Riemann Programming Language, ISBN: 978-1906717605). The main motivation at that time for the name was the metaphorical correspondence between multi-valued functions represented by Riemann surfaces and software defects as alternative branches of computation. Since the significant development of pattern-oriented software diagnostics, introduction of network (Pattern-Oriented Network Trace Analysis, ISBN: 978-1908043580) and performance analysis (Volume 8a) pattern languages and patterns-based root cause analysis methodology (Volume 9a) we now make Riemann Programming Language an optional coding complement to Riemann Root Cause Analysis Pattern Language. The latter includes diagnostic analysis pattern languages for trace analysis (Software Trace and Log Analysis: A Pattern Reference) and memory analysis (Encyclopedia of Crash Dump Analysis Patterns: Detecting Abnormal Software Structure and Behavior in Computer Memory) developed by Software Diagnostics Institute including structural memory patterns (Volume 5) in the context of general log analysis (Special and General Trace and Log Analysis, Volume 8b). We can now consider another analogy with multi-valued functions where the same general diagnostic patterns (Pattern! What Pattern? Volume 8b) in a memory dump or log can be generated by different source code. Riemann RCA Pattern Language facilitates the transformation of software narrative artifacts into much shorter analysis narratives through the process of articoding (Coding and Articoding, Volume 8b). The resulting analysis artifacts can be programmatically processed to generate diagnostic, troubleshooting and debugging configurations, classes and functions, frameworks and plugins, components and nodes. The following diagram describes this process:

The Riemann programming language should not be confused with Riemann monitoring system which was named and developed later elsewhere by a different group of people and which is about collecting events and not about their collective analysis using pattern-oriented analysis methodology developed by Software Diagnostics Institute. Regarding event monitoring, Software Diagnostics Institute also develops platform-independent software trace and log acquisition patterns (Volume 7) for better use of various monitoring systems.

Problem Solving as Code

We introduce Problem Solving as Code as a process of developing, managing, and provisioning problem-solving methods and tools. Some problem-solving methodologies such pattern-oriented problem-solving developed by Software Diagnostics Institute as a part of Diagnostics Science require constantly evolving pattern catalogs which can be stored in version control systems. For example, pattern-oriented software problem-solving (Introduction to Pattern-Driven Software Problem Solving, ISBN: 978-1908043177) involves pattern-oriented problem description analysis (Software Problem Description Patterns, Volume 7) and software execution memory (Memory Acquisition Pattern Catalog, Volume 7) and trace (Trace Acquisition Pattern Catalog, Volume 7) artifact acquisition, pattern-driven (Introduction to Pattern-Driven Software Diagnostics, ISBN: 978- 1908043382) and pattern-based (Introduction to Pattern-Based Software Diagnostics, ISBN: 978- 1908043498) software diagnostics (Pattern-Based v. Pattern-Driven Software Diagnostics, Volume 7) (including forensics, Pattern-Oriented Software Forensics: A Foundation of Memory Forensics and Forensics of Things, ISBN: 978-1908043696), the patterns-based root cause analysis (Patterns-Based Root Cause Analysis Methodology, Volume 9a), and pattern-oriented debugging process (Pattern-Oriented Debugging Process, Volume 8a) which introduced design methodology to debugging (Analysis, Architectural, Design, Implementation and Usage Debugging Patterns, Volume 7). In addition to general problem patterns and problem analysis patterns, there are concrete problem and problem analysis patterns (Pattern! What Pattern? Memory Dump Analysis Anthology, Volume 8b) where concrete problems are constantly changing (traditional problem repositories). PSaC (“Problem Sack”) allows using declarative and imperative problem-solving configurations tailored for specific problem domains or specific systems and products by customizing pattern catalogs. Specific problem artifact types may require specialized tools and configuration so they can also be designed, developed, managed, and provisioned. For example, pattern-oriented problem solving includes DebugWare (DebugWare patterns, Memory Dump Analysis Anthology, Volume 2) and DiagWare design patterns.

Dia|gram Graphical Diagnostic Analysis Language

One of the current Software Diagnostics Institute projects is the development of Dia|gram graphical language for pattern-oriented software diagnostics, forensics, prognostics, root cause analysis and debugging. It combines the best features from:

1. Visual Dump Objects: Graphical Notation for Memory Dumps (Volume 3);
2. STDiagrams: Software Trace Diagrams (Volume 7);
3. Visual compression of software traces and logs (including “bird’s eye view” of software traces), first introduced in Characteristic Message Block (Volume 4) trace and log analysis pattern;
4. Minimal Trace Graphs, first introduced in Activity Region (Volume 4) trace and log analysis pattern. Numerous examples can be found in Accelerated Windows Software Trace Analysis training course reference and Software Trace and Log Analysis: A Pattern Reference book;
5. Minimal Stack Trace Diagrams, first introduced in Constant Subtrace (Volume 9b) memory analysis pattern.

The purpose of Dia|gram language is twofold:

• To provide a succinct presentation and visualization of software execution state, artifacts, distribution of problem patterns, problem analysis patterns and their relationship (Volume 8b);
• To communicate pattern-oriented software diagnostic analysis results.

Additionally, Dia|gram may be used for presentation and analysis of higher-order pattern narratives (Higher-Order Pattern Narratives, Analyzing Diagnostic Analysis, Volume 8a).

Software Diagnostics Institute also proposes the UML profile for Software Diagnostics with additional diagram types: artifact acquisition map, activity backtrace, and implementation internals. This work is only started, and more will be presented in subsequent articles.

Software Diagnostics Services plans to include Dia|gram in its forthcoming Advanced Software Trace and Log Analysis training course.

Iterative Pattern-Oriented Root Cause Analysis

When we introduced A.P.M. patterns-based root cause analysis methodology (Artifacts. Patterns. Mechanisms., Patterns-Based Root Cause Analysis Methodology, Volume 9a), it may have made an impression of a waterfall-type process with some iterations between artifact collection and diagnostic analysis when collected artifacts are not good. However, software post-construction problem solving is usually iterative, with memory dumps and software logs collected again and again after the preliminary root cause analysis.

To illustrate the iterative nature of the process we first name its stages as Artifact Acquisition for Artifacts, Artifact Analysis for Patterns (diagnostics), and Analysis of Analysis for Mechanisms (root cause analysis):

Now we rearrange these AA stages:

After the preliminary root cause analysis (Analysis of Analysis) we may need to gather more artifacts for further diagnostics and more precise RCA, and this is reflected in more focused stages:

Theoretical Software Diagnostics and Education

After writing so much about software diagnostics, we introduce its abstract generalizing principles of pattern orientation and systems thinking as Theory of Software Diagnostics. We were thinking about the importance of theory for quite some time until we got acquainted with the work of Leo Klejn who coined a term “theoretical archaeology.” Then we also decided to coin the similar term for software meta-diagnostics since we compiled two books as guides to software diagnostics principles irrespective of software platforms, vendors, and their software products: Software Diagnostics (Software Diagnostics: The Collected Seminars, ISBN: 978-1908043641) and Principles of Memory Dump Analysis (Principles of Memory Dump Analysis: The Collected Seminars, ISBN: 978- 1906717667) and plan to publish a compilation of related theoretical articles (Theoretical Software Diagnostics, ISBN-13: 978-1-908043-98-6). Looking for the development of theoretical archaeology as guidance makes sense because it emerged recently in contemporary times and also deals with artifacts, historical reconstruction, and time- and memory-related issues, albeit of a different nature. While working on theoretical foundations and principles for many years, we had to learn theories, ideas, and metaphors of other disciplines used in software diagnostics that we call software para-diagnostic theories by analogy with para-archaeological (coined by Klejn) theories such as history, sociology, linguistics. In his book “Introduction to Theoretical Archaeology: Meta-archaeology,” (ISBN 5-9259-0039-1, Extended Russian edition of Klejn L. S. Metaarchaeology) Klejn made a few remarks on the required theoretical education. We would like to reformulate them in relation to theoretical software diagnostics:

• Very few people do theory because theoretical thinking requires broad education and polymath knowledge across many disciplines. We found that:

• Computer science and software engineering education helps in the practical side of software diagnostics but is not enough;
• Knowledge of university-level mathematics and natural science education help in understanding of technical diagnostics but is not enough;
• Knowledge of the principles of medical diagnostics helps because pattern-oriented facet of theoretical software diagnostics is partially based on medical metaphors;
• Knowledge of semiotics helps in understanding of the role of signs in theoretical software diagnostics;
• Knowledge of philosophy helps in deeper understanding of foundational aspects of theoretical software diagnostics such as the nature of problems, their phenomenology, meaning, and understanding;

• Humanities education (analysis of human-made artifacts) is very important since software diagnostics is also based on artifact analysis.

• Such education is needed from earlier up and in addition to computers and coding should also include history, philology, narratology, and literary theory.

• In summary, broad reading is required to get acquainted with diagnostics expertise in various domains of human activity.