M
Macrofunction is a single semantic unit of several trace messages where individual messages serve the role of microfunctions. We borrowed this idea and distinction from functionalist linguistics. An example would be a software trace fragment where messages log an attempt to update a database:
# Module PID TID Time Message
[...]
42582 DBClient 5492 9476 11:04:33.398 Opening connection
[...]
42585 DBClient 5492 9476 11:04:33.398 Sending SQL command
[...]
42589 DBServer 6480 10288 11:04:33.399 Executing SQL command
[...]
42592 DBClient 5492 9476 11:04:33.400 Closing connection
[...]
These Macrofunctions need not be from the same ATID (Glued Activity) in the traditional sense like in the example above unless we form Adjoint Threads from certain fragments like “DB.”
Based on ideas of Roman Jakobson76 about “marked” and “unmarked” categories, we propose this pattern that groups trace messages based on having some feature or property. For example, marked messages may point to some domain of software activity such related to functional requirements and, for this reason, may help in troubleshooting and debugging. Unmarked messages include all other messages that don’t say anything about such activities (although they may include messages pointing to such activities indirectly we unaware of) or messages that say explicitly that no such activity has occurred. We can even borrow a notation of distinctive features77 from phonology78 and annotate any trace or log after analysis to compare it with Master Trace, for example, compose the following list of software trace distinctive features:
session database queries [+]
session initialization [-]
socket activity [+]
process A launched [+]
process B launched [-]
process A exited [-]
[...]
Here [+] means the activity is present in the trace and [-] means the activity is either undetected or definitely not present. Sometimes a non-present activity can be a marked activity corresponding to all-inclusive unmarked present activity (see, for example, No Activity pattern).
When reading and analyzing software traces and logs, we always compare them to Master Trace. Other names for this pattern borrowed from narrative theory include Metatrace, Canonical Trace, or Archetype. When we look at the software trace from a system we either know the correct sequence of Activity Regions, expect certain Background and Foreground Components, Event Sequence Order, or mentally construct a model based on our experience and Implementation Discourse. For the latter example, software engineers internalize software master narratives when they construct code and write tracing code for supportability. For the former example, it is important to have a repository of traces corresponding to Master Traces. Such a repository helps in finding deviations after Bifurcation Point. Consider such comparisons similar to regression testing when we check the computation output against the expected prerecorded sequence.
Message Annotations analysis pattern was on our list for years, so it’s time to add it to trace and log analysis pattern catalog. We also allow several annotations per trace message done at different times and annotating different p arts depending on the analysis flow (the topic of the next analysis pattern). When attached to log messages, annotations form their Adjoint Thread of Activity. However, sorted by their annotation time or sequence, we get a different trace that we call Annotation Trace (an example of analysis narrative79). We can apply all relevant analysis patterns to both traces. A sketch of this pattern is depicted in the following diagram:
Sometimes, when we find Anchor Message related to our problem description (for example, a COM port error) we are interested in its evolution throughout a software narrative:
# PID TID Message
[...]
126303 5768 1272 OpenComPort returns Status = 0x0
[...]
231610 3464 1576 OpenComPort returns Status = 0x0
[...]
336535 5768 4292 OpenComPort returns Status = 0×0
[...]
423508 5252 2544 OpenComPort returns Status = 0xc000000f
[...]
531247 5768 5492 OpenComPort returns Status = 0xc000000f
[...]
639039 772 3404 OpenComPort returns Status = 0xc000000f
[...]
Then we can check activity between changes.
The following diagram illustrates this analysis pattern:
In some cases, it is useful to consider a message context: a set of surrounding messages having some relation to the chosen message:
One of the powerful trace analysis techniques is using Adjoint Threads of Activity to filter various linear message activities (as a generalization of Thread of Activity). Such filtered activities can then be analyzed either separately (Sheaf of Activities) or together, such as a new pattern we introduce here. If we identify parallel ATIDs (ATID is Adjoint TID80) and see that one covers the other, we can then make a hypothesis that they are Intra-Correlated. Here is a graphical example of Periodic Message Block largely composed from various Error Messages that covers periodic Discontinuities from another ATID (we can also consider the latter as Periodic Message Blocks consisted from Silent Messages):
This pattern is analogous to a cover81 in topology.
We can apply Message Set trace and log analysis pattern to itself and build a multilevel Message Directory. The pattern name come from the analogy with memory management page tables, where we can select every 10th message to build a smaller Message Set, and then select every 10th message from it to build another Message Set, and so on. When looking at a higher level Message Set, we can select a trace message and then look at its Message Context in the lower level Message Sets:
Message Directory is similar to Trace Frames analysis pattern but is fine-grained, and more individual message-oriented than the latter analysis pattern. However, the former pattern can be used to implement the latter.
We factored out Anchor Messages example of Message Interleave into this pattern. It covers superposition of different Anchor Messages, for example, process launch and exit, or DLL load and unload:
Most of the time software trace messages coming from the same source code fragment (PLOT82) contain invariant parts such as function and variable names, descriptions, and mutable parts such as pointer values and error codes. Message Invariant is a pattern useful for comparative analysis of several trace files where we are interested in message differences. For example, in one troubleshooting scenario, certain objects were not created correctly for one user. We suspected a different object version was linked to a user profile. We recorded separate application debug traces for each user, and we could see the version 0×4 for a problem user and 0×5 for all other normal users:
# Module PID TID Message
[...]
2782 ModuleA 2124 5648 CreateObject: pObject 0×00A83D30 data ([...]) version 0×4
[...]
# Module PID TID Message
[...]
4793 ModuleA 2376 8480 CreateObject: pObject 0×00BA4E20 data ([...]) version 0×5
[...]
Now we come to the trace and log analysis pattern that we call Message Pattern. It is an ordered set of messages from Thread of Activity or Adjoint Thread of Activity having Message Invariants that can be used for matching another ordered set of messages in another (Inter-Correlation) or the same trace or log (Intra-Correlation). A typical Message Pattern from one of our trace and log analysis sessions is depicted in the following diagram:
Often, especially for large software logs, we need to select messages based on some criteria be it a set of Error Messages, a set of messages containing Basic Facts, or some other predicate. Then we can use selected messages from that message set as Anchor Messages or reverse Pivot Messages as an aid in further analysis.
The message stream can be considered as a union of Message Spaces. A message space is an ordered set of messages preserving the structure of the overall trace. Such messages may be selected based on memory space they came from or can be selected by some other general attribute, or a combination of attributes and facts. The differences from Message Set is that Message Space is usually much larger (with large scale structure) with various Message Sets extracted from it later for fine-grained analysis. This pattern also fits nicely with Adjoint Spaces. Here’s an example of the kernel and managed spaces in the same CDF / ETW trace from Windows platform where we see that kernel space messages came not only from System process but also from other process contexts:
In the context of general traces and logs83 such as debugger logs, separate Message Space regions may be linked (or “surrounded”) by Interspace.
So far, we have been discussing trace analysis patterns related to the execution of a particular software version. However, software code changes and also its tracing and logging output: from large-scale changes where components are replaced to small scale code refactoring affecting message structure and format. On a software narratological level, this corresponds to a narrative about a software trace or log, its evolution. Such an analysis pattern is different from Master Trace pattern where the latter is similar to what Metanarrative84 is usually meant in narratology: a master or grand idea, an expected trace if all functional requirements were correctly identified and implemented during software construction, and non-functional ones met during software execution.
Trace messages may correspond to specific implementation code such as recording the status of an operation, dumping data values, printing errors, or they may correspond to higher levels of software design and architecture, and even to use case stories. We call such messages Milestones by analogy with project management85. Alternative names can be Chapter Messages, Summary Messages, Checkpoints, or Use Case Messages. These are different from Macrofunctions, which are collections of messages grouped by some higher function. Milestone messages are specifically designed distinct trace statements:
They can also be a part of Significant Events, serve the role of Anchor Messages, and be a part of Basic Facts and Vocabulary Index.
If we run software with its default configuration and no interaction (input data), we get its Minimal Trace:
Such traces may have their own Master Trace. Also, Minimal Trace is a specific Use Case Trail. Metaphorically, they can be considered as minimal surfaces86.
When we do not see expected trace statements, we wonder whether the component was not loaded, its container ceased to exist or simply it was not selected for tracing. In many support cases, there is a trade-off between tracing everything and the size of trace files. Customers and engineers usually prefer smaller files to analyze. However, in the case of predictable and reproducible issues with short duration, we can always select all components or deselect a few (instead of selecting a few).
In Discontinuity pattern, the possibility of a sudden and silent gap in trace statements could happen because not all necessary components were selected for tracing.
Sometimes, in cases when the missing component was selected for tracing, but we do not see any trace output from it, other module traces can give us an indication, perhaps showing the load failure message. For example, Process Monitor tracing done in parallel can reveal load failures.
The following diagram illustrates this analysis pattern:
Some tracing architectures, especially the ones that intercept API calls by filtering or hooking, may log synchronous requests by remembering to write done return result in the same trace message later on when the response is available after the wait. If such data is still not available in the log or trace, it may point to some blocked request for which another software execution artifact analysis (such as memory dump analysis) is necessary. In some cases, the analysis of the corresponding Fiber Bundle stack trace may point to Blocking Module87 or the involvement of file system filters (Stack Trace88). This analysis pattern that we call Missing Data is illustrated in the following diagram:
Sometimes the absence of messages, for example, errors and exceptions, may save time during troubleshooting and debugging by pointing to what was not happening and provide additional insight. For example, in the picture below, we see the same exceptions in the new and old incidents. However, in the old incident, we see another exception that was linked to one unavailable server in distributed broker architecture. For this reason, we can assume provisionally that all servers were operational when the new incident happened.
Missing Message pattern is different from Missing Component pattern where the latter may point to the component that was not loaded or executed, or simply that it was not selected for tracing.
In addition to Declarative Trace, we have code statements that may be intercepted by external API tracing tools (CreateFile, CloseHandle):
In the sample code above, we have different logs resulted from Declarative Trace (DebugLog, OutputDebugString):
Also, we have a log saved by an external tracing tool (for example, Process Monitor) that includes our API calls:
All such trace-generating source code statements form Moduli Trace as soon as they are executed:
Such a trace can also be analyzed using trace and log analysis patterns like other trace types. We took the idea of this analysis pattern from moduli spaces89 in mathematics that parametrize other spaces.
Often, when we look at software trace fragments, we recognize certain motifs such as client-server interaction, publisher-subscriber notifications, database queries, plugin sequence initialization, and others. The idea of this pattern name comes from motives90 in mathematics. It is different from Master Trace, which corresponds to a normal use-case or working software scenario and may contain several Motifs as it usually happens in complex software environments. On the other side of the spectrum, there are software narremes (basic narrative units91) and Macrofunctions (single semantic units). Motifs help to further bridge the great divide between software construction and software diagnostics with software narremes corresponding to implementation patterns, Macrofunctions to design patterns, and Motifs to architectural patterns although the overlap between these categories is possible.
We can “integrate” trace message stream into another, smaller trace. By analogy with motivic integration92 in contemporary mathematics, we call this analysis pattern Motivic Trace. There can be border cases where the whole trace is reduced to one message, or every message is associated with a different message (perhaps shorter or a number). Message Sets that are integrated into Motivic Trace can be completely different (for example, based on Motives) in comparison with Quotient Trace where we reduce Message Sets that have the same common attribute.
