PART 3: Software Trace Analysis Patterns

Data Selector

Data Selector is a variant of Inter-Correlation (Volume 4) trace analysis pattern where we use data found in one trace to select Message Set (Volume 7) or Adjoint (Volume 5) Thread of Activity (Volume 4) in another trace. This analysis activity is depicted in the following picture where we have a client log and corresponding server log. In the server log, we have entries for many client sessions. To select messages corresponding to our client session we use some data attribute in the client trace, for example, the username, and Linked Messages (Volume 7) analysis pattern to find one of the messages in the server log that contains the same username. Then we find out which user session it belongs to and form its Adjoint Thread (Volume 5):

This pattern is different from Identification Messages (Volume 9a) where we don’t even know the object that emitted trace messages. In Data Selector case we know in principle what kind of messages we are looking for. We just need to select among many alternatives.

Declarative Trace

The trace statements in source code can be considered as Declarative Trace by analogy with variable declaration and definition in programming languages such as C and C++. Declaration of the variable doesn’t mean that the variable will be actually used. Some declared variables such as arrays will actually expand in memory when used (as in .bss sections). The same is with trace messages from Declarative Trace. Some of them will not appear in the actual software execution trace, and some will be repeated because of loops and multiple code reentrance. However, Declarative Traces are useful for studying the possibilities of tracing and logging design, implementation, and coverage (for example, Sparse Trace, Volume 7). Some trace analysis patterns are also applicable for Declarative Traces such as Message Sets (Volume 7) and Bifurcation Points (Volume 4) among different source code versions. This is illustrated in the following picture:

Trace Extension

Trace Extension is an obvious log analysis pattern that is about trace messages that refer to some other trace or log that may or may not exist. Sometimes, there can be instructions to enable additional tracing that is not possible to cover by the current trace source. We have seen this in some trace statements from .NET Exception Stack Traces (Volume 4).

Fourier Activity

Sometimes we have trace and log messages that appear with certain time-frequency throughout all log or specific Thread of Activity (Volume 4). Such frequencies may fluctuate reflecting varying system or process performance. Analyzing trace areas where such messages have different Time Deltas (Volume 5) may point to additional diagnostic log messages useful for root cause analysis. The following minimal trace graph depicts the recent log analysis for proprietary file copy operation where the frequency of internal communication channel entry/exit Opposition Messages (Volume 7) was decreasing from time to time. Such periods were correlating with increased time intervals between “entry” and “exit” messages. Analysis of messages between them revealed additional diagnostic statements missing in periods of higher frequency and corresponding Timeouts (Volume 8b) adding up to overall performance degradation and slowness of copy operation.

Additional analysis of Data Association (Volume 7) in a different message type between available communication buffers and the total number of such buffers revealed significant frequency drop during constant Data Flow (Volume 7) of zero available communication buffers:

We call this analysis pattern Fourier Activity by analogy with Fourier series in mathematics. This pattern is for individual message types and can also be considered a fine-grained example of Statement Current (Volume 4) and Trace Acceleration (Volume 5) analysis patterns which can be used to detect areas of different frequencies in individual Fibers (Adjoint Threads of Activities, Volume 5, formed from the same Thread of Activity).

Fiber of Activity

When using complex trace and log analysis patterns such as Fourier Activity (Volume 9b) we may be first interested in selecting all instances of a particular message type from specific Thread of Activity (Volume 4) and then look for Time Deltas (Volume 5), Discontinuities (Volume 4), Data Flow (Volume 7), and other patterns. We call this analysis pattern Fiber of Activity by the analogy of fibers (lightweight threads) since the individual thread execution flow is “co-operative” inside, whereas threads themselves are preempted outside. The following diagram from Fourier Activity analysis pattern example illustrates the concept by showing three fibers:

This analysis pattern is different from trace-wide Sheaf of Activities (Volume 7) where the latter is about selecting messages as Adjoint Threads of Activity (Volume 5) which may span several processes and threads.

Missing Data

Some tracing architectures, especially the ones that intercept API calls by filtering or hooking, may log synchronous requests by remembering to write done return result in the same trace message later on when the response is available after the wait. If such data is still not available in the log or trace, it may point to some blocked request for which other software execution artifact analysis (such as memory dump analysis) is necessary. In some cases, the analysis of the corresponding Fiber Bundle (Volume 7) stack trace may point to Blocking Module (Volume 6) or the involvement of file system filters (Stack Trace, Volume 8a). This analysis pattern that we call Missing Data is illustrated in the following diagram:

Message Pattern

Now we come to the trace and log analysis pattern that we call Message Pattern. It is an ordered set of messages from Thread of Activity (Volume 4) or Adjoint Thread of Activity (Volume 5) having Message Invariants (Volume 6) that can be used for matching another ordered set of messages in another (Inter-Correlation, Volume 4) or the same trace or log (Intra-Correlation, Volume 3). A typical Message Pattern from one of our own trace and log analysis sessions is depicted in the following diagram:

Activity Theatre

In addition to Message Patterns (Volume 9b), there are higher level patterns of specific activities and Motives (Volume 7). Such activities may or may not coincide with specific components (modules) because they may be grouped based on implementation messages, software internals semantics and not on architectural and design entities (as in Use Case Trail analysis pattern, Volume 8a). Moreover, the same components may “play” different activity roles. Once assigned, Activity Theatre “scripts” can be compared with “scripts” from other traces and logs (Inter-Correlation, Volume 4) or different parts of the same log (Intra-Correlation, Volume 3). This pattern is illustrated in the following diagram:

Small DA+TA

Recently we performed the diagnostic analysis of a software incident where certain functionality was not available to users and provided the report based on analysis patterns such as Focus of Tracing (Volume 6) and Opposition Messages (Volume 7). We also conjectured some hypotheses explaining the observed abnormal behavior. However, in the end, the problem was solved not by the analysis of a lengthy software execution log but by looking at the small configuration INI file where not working functionality was simply disabled in one line:

EnableFunctionality = 0

Even before that analysis, we were thinking about the importance of Small DA+TA such as configuration files and registry details that can be considered as general software traces (Volume 8b). Here DA+TA means Dump Artifact + Trace Artifact and Big DA+TA refers to software execution memory dump artifacts and trace artifacts that can be really huge. The analysis pattern is illustrated in the following diagram where we see no difference between working and non-working scenarios due to insufficient trace coverage (Sparse Trace, Volume 7):

Surveyor

Sometimes, the presence of some messages in a trace or log shows that some other tracing or logging tool was running or that some process was also doing tracing. We call this analysis pattern Surveyor. Such discovered tracing may not be related to the trace we are looking at (compare to Trace Extension, Volume 9b) but may help with finding additional traces in the system as illustrated in the following diagram:

Quotient Trace

In Adjoint Message (Volume 9a) analysis pattern description, we mentioned compressing message sequences having the same message attribute into one message. Considering the trace as “topological” space and message attribute as “equivalence” relation we introduce Quotient Trace analysis pattern by analogy with quotient space in topology. By endowing message sequences having the same attribute with some “metric” such as cardinality of Message Set (Volume 7) we can also visually distinguish resulted quotient messages if they have the same attribute but from different sequences at different times. All this is illustrated in the following diagram: