7.2.1 Getting the Data

In many cases our data will be text (e.g., source code) we can directly feed to the standard input of a tool. If this is not the case, we need to adapt our data. If we are dealing with object files or executable programs, we will have to use a tool such as nm (Unix), dumpbin (Windows), or javap (Java) to dig into them. We examine these approaches in Sections 7.4.2 and 7.4.4. If we are working with files grouped into an archive, then a command such as tar, jar, or ar will list the archive’s contents. In addition, the ldd command will print shared library dependencies associated with Unix executables, object files, and libraries. If our data comes from a (potentially large) collection of files stored on locally accessible storage find can locate those that interest us. Here is how we would use the find command to list the (header) files residing in the directory /usr/include.²

On the other hand, to get our data over the Web, we can use wget or curl (see Section 7.5.1). We can also use dd (and the special file /dev/zero), yes, or jot to generate artificial data, perhaps for running a quick test or a benchmark. Finally, if we want to process a compiler’s list of error messages, we will want to redirect its standard error to its standard output; the incantation 2>&1 will do this trick.

There are many other cases we have not covered here: relational databases, version control systems (see Section 7.5), mailing lists, issue management systems, telemetry data, and so on. A software system’s issue management system (bugs) database can provide insights regarding a product’s maturity and the adequacy of the resources devoted to it. Issues are often coupled with software changes, allowing even more detailed analyses to be performed. A dynamic view of a system’s operation can be obtained by analyzing software telemetry data. This can include precise user interaction metrics, crash dump reports [23], and server logs. Always keep in mind that we are unlikely to be the first ones who need the application’s data converted into a textual format; therefore, someone has probably already written a tool for that job. For example, the Outwit tool suite [24]³ can convert into a text stream data coming from the Windows clipboard, an odbc source, the event log, or the Windows registry.

7.2.2 Selecting

Given the generality of the textual data format, in most cases we will have on our hands more data than what we require. We might want to process only some parts of each row, or only a subset of the rows. To select a specific column from a line consisting of elements separated by white space or another field delimiter, we can use awk with a single print $n command. If our fields are of fixed width, we can separate them using cut. And, if our lines are not neatly separated into fields, we can often write a regular expression for a sed substitute command to isolate the element we want.

The workhorse for obtaining a subset of the rows is grep. We can specify a regular expression to get only the rows that match it, and add the --invert-match⁴ flag to filter out rows we do not want to process.

Here is how we could use grep to list lines in the Freebsd kernel source code file vfs_subr.c containing the XXX sequence, which is commonly used to flag questionable code. The first part of the fetch-selection pipeline uses curl to fetch the corresponding file from the Freebsd repository. The backslash at the end of the line indicates that the line is continued on the one below, containing the url where the file resides. The | (pipeline) symbol specifies that the output of curl (the vfs_subr.c file’s contents) will be sent for further processing to the command following it, which is grep.

We can use the grep flags --files-with-matches and --files-without-match to obtain only the names of files that contain (or do not contain) a specific pattern. We can run fgrep with the --file flag if the elements we are looking for are fixed strings stored in a file (perhaps generated in a previous processing step). If our selection criteria are more complex, we can often express them in an awk pattern expression. We will often find ourselves combining a number of these approaches to obtain the result that we want. For example, we might use grep to get the lines that interest us, grep --invert-match to filter-out some noise from our sample, and finally awk to select a specific field from each line.

Many examples in this chapter use awk for some of their processing steps. In general, awk works by applying a recipe we give it as an argument on each line of its input. This recipe consists of patterns and actions; actions without a pattern apply to all input lines, and a pattern without an action will print the corresponding line. A pattern can be a /-delimited regular expression or an arbitrary boolean expression. An action consists of commands enclosed in braces. Lines are automatically split into space-separated fields. (The -F option can be used to specify arbitrary field delimiters.) These fields are then available as variables named $ n. For example, the following shell command will print the names of header files included by the C files in the current directory.

7.2.3 Processing

Data processing frequently involves sorting our lines on a specific field. The sort command supports tens of options for specifying the sort keys, their type, and the output order. Having our results sorted we then often want to count how many instances of each element we have. The uniq command with the --count option will do the job here; often we will post-process the result with another instance of sort, this time with the --numeric flag specifying a numerical order, to find out which elements appear most frequently. In other cases, we might want to the compare results between different runs. We can use diff if the two runs generate results that should be similar (perhaps we are comparing two versions of the file), or comm if we want to compare two sorted lists. Through comm we can perform set intersection and difference operations. To piece together results coming from unrelated processing steps based on a key, we can first sort them and then apply the join command on the two lists. We can handle more complex tasks using, again, awk.

Here are the steps of how to build a pipeline that generates a list of header files ordered by the number of times they are included. First, use grep to obtain a list of include directives.

Then, use awk to get from each line the included file name, which is the second field. While at it, we can replace the original grep with an awk selection pattern.

Our next step is to sort the file names, in order to bring the same ones together, so that they can be counted with uniq.

We then use uniq to count same consecutive lines (file names).

The final step involves sorting the output again, this time in reverse numerical order, in order to obtain a list of header file names in a descending order according to the number of times they occurred in the source code.

7.2.4 Summarizing

In many cases the processed data are too voluminous to be of use. For example, we might not care which symbols are defined with the wrong visibility in a program, but we might want to know how many there are. Surprisingly, many problems involve simply counting the output of the processing step using the humble wc (word count) command and its --lines flag. Here again is the preceding example, this time counting the number of lines containing the characters XXX.

If we want to know the top or bottom 10 elements of our result list, we can pass our list through head or tail. To format a long list of words into a more manageable block that we can then paste in a document, we can use fmt (perhaps run after a sed substitution command tacks on a comma after each element). Also, for debugging purposes we might initially pipe the result of intermediate stages through more or less, to examine it in detail. As usual, we can use awk when these approaches do not suit us; a typical task involves summing up a specific field with a command such as sum += $3. In other cases, we might use awk’s associative arrays to sum diverse elements.

7.2.5 Plumbing

All the wonderful building blocks we have described are useless without some way to glue them together. For this we will use the Bourne shell’s facilities. First and foremost comes the pipeline (|), which allows us to send the output of one processing step as input to the next one, as we saw in the preceding examples. We can also redirect output into a file; this is done by ending our command with the > file-name construct. In other cases, we might want to execute the same command with many different arguments. For this we will pass the arguments as input to the xargs command. A typical pattern involves obtaining a list of files using find and processing them using xargs. Commands that can only handle a single argument can be run by xargs if we specify the --max-args=1 option. If our processing is more complex, we can always pipe the arguments into a while read loop. (Amazingly, the Bourne shell allows us to pipe data into and from all its control structures.) When everything else fails, we can use a couple of intermediate files to juggle our data.

Note that by default the Unix shell will use spaces to separate command-line arguments. This can cause problems when we process file names that contain spaces in them. Avoid this by enclosing variables that represent a file name in double quotes, as in the following (contrived) example that will count the number of lines in the Java files residing in the directories under org/eclipse.

When using find with xargs, which is more efficient than the loop in the preceding example, we can avoid the problem of embedded spaces by using the respective arguments -print0 and --null. These direct the two commands to have file names separated with a null character, instead of a space. Thus, the preceding example would be written as

7.3.1 Heuristics

Heuristics allow us to easily obtain rough-and-ready metrics from the code. The main advantage of heuristics in source code analysis is that they are easy, often trivial, to implement. They can therefore often be used as a quick way to test a hypothesis. In most cases, the use of heuristics entails the use of regular expressions and corresponding tools, such as the family of the Unix grep programs, to obtain measures that are useful, but not 100% accurate. For instance, the following command will display the number of top-level classes defined in a set of Java files.

The heuristic employed here is based on the assumption that the word class appears in the beginning of a line if and only if it is used to define a top-level class. Similarly, the number of subclasses defined in a set of files can be found through the following command.

Again, the preceding command assumes that the word extends is only used to refer to a subclass’s base class. For example, the count can be set off by the word extends appearing in strings or comments. Finally, if the files were located in various folders in a directory tree, we could use the grep’s --recursive flag, instructing it to traverse the directory tree starting from the current directory (denoted by a dot). An invocation of awk can then be used to sum the counts (the second field of the colon-separated line).

The preceding command assumes that the keyword class always appears at the beginning of a line, and that no other files that might contain lines beginning with the word class are located within the directory hierarchy.

7.3.2 Lexical Analysis

When more accuracy is required than what a heuristic can provide, or when the analysis cannot be easily expressed through a heuristic, flow-blown lexical analysis has to be performed. This allows us to identify reserved words, identifiers, constants, string literals, and rudimentary properties of the code structure. The options here include expressing the lexical analyzer as a state machine or creating code with a lexical analyzer generator.

7.3.3 Parsing and Semantic Analysis

Parsing and semantic analysis [26] is required when we want to extract more sophisticated measures from the code, involving, for instance, the scoping of identifiers, the handling of exceptions, and class hierarchies. Most modern languages are large complex beasts and therefore this form of processing is not for the faint-hearted. The effort involved can easily require writing tens of thousands of lines of code. Therefore, if this level of analysis is required it is best to adapt the code of an existing compiler. Compilers for most languages are available as open source software, and can therefore can be modified to perform the requisite analysis.

An interesting case is the llvm compiler infrastructure [27] and in particular its Clang front-end, which can be used as a library to parse and analyze C-like languages, such as C, C++, and Objective C. For instance, we can build an analyzer that will print a C program’s global variable declarations in about 100 lines of C++ code.⁵

7.3.4 Third-Party Tools

A final option to analyze source code is to piggy-back third-party tools that analyze code. Here are some tools and ideas on how to use them.

The CScout program is a source code analyzer and refactoring browser for collections of C programs [28]. It can process workspaces of multiple projects (think of a project as a collection of C source files that are linked together) mapping the complexity introduced by the C preprocessor back into the original C source code files. CScout takes advantage of modern hardware (fast processors and large memory capacities) to analyze C source code beyond the level of detail and accuracy provided by current compilers, linkers, and other source code analyzers. The analysis CScout performs takes into account the identifier scopes introduced by the C preprocessor and the C language proper scopes and namespaces. After the source code analysis CScout can process sophisticated queries on identifiers, files, and functions, locate unused or wrongly-scoped identifiers, and compute many metrics related to files, functions, and identifiers. Figure 7.1 illustrates the metrics collected for functions.

CCFinderX⁶ is a tool that detects duplicated code fragments in source code written in many modern programming languages. The tool is a redesign of CCFinder [29], which has been used for research published in many papers. The command-line version of the tool will print its results as a text file.

The output file format of CCFinderX is simple, but not trivial. Its first section lists for each file that has been analyzed its numerical identifier, its path, and the number of tokens it contains. Here is an excerpt corresponding to the Linux kernel.

A list of detected code clones then follows. Each line contains the clone’s identifier, followed by a pair of source code clone specifications. Each one has the file identifier, the beginning token, and the end token of the cloned code.

In the following example we see code cloned within the same file (microcode.c—clone 7329), as well as code cloned between different files (e.g., cpuid.c and msr.c—clone 6981).

The generated file can be further analyzed to derive additional measures. As an example, the following Perl program when given as input the base name of a CCFinderX result file will print the percentage of cloned tokens in it. A high percentage of cloning can often lead to higher maintenance costs, because fixes and enhancements need to be carefully duplicated in multiple places.

General-purpose tools can often be just as helpful as the specialized ones we have seen. If we want to perform some processing on comments in C, C++, or Objective-C code, then the gcc version of the C preprocessor can help us. Consider a case where we want to count the number of comment characters in a source code file. Preprocessing the file with the -fpreprocessed flag will remove the comments, but won’t perform any other expansion. Thus, subtracting the number of characters in the file with the comments removed from the original number will give us the number of comment characters. The following sh code excerpt will print the number of comment characters in file.c.

We can also pass the -H flag to the C preprocessor in order to obtain a list of included header files. This output can, for instance, then be used to map code reuse patterns. Here is some representative output. (The dots at the beginning of each line indicate nested include levels, and can be used to study a project’s module layering.)

Another useful family of general-purpose tools we can repurpose for source code analysis are documentation generators, such as Doxygen and Javadoc. These parse source code and documentation comments to create code reference documents. The simplest way to use these tools is to analyze the resulting html text. The text’s structure is simpler than the corresponding code, and, in addition, it may contain data that would be difficult to get from the original code. The trick in this case is to look at the generated html code (right-click—This Frame—View Source in many browsers) to determine the exact pattern to search for. For example, the following shell code will go through the Java development kit html documentation to count the number of methods that are declared to implement some interface (7752 methods in total).

If the generated documentation does not contain the information we want, we can extend Javadoc through custom so-called doclets. These have a method that is called after Javadoc has processed the source code. The method gets as an argument a document tree of the code’s element, which can then be easily processed to extract and print the results we want. As an example, the UMLGraph system uses this approach to create uml diagrams out of Java code [30].

Regarding the analysis of the code’s adherence to some style conventions, a useful approach is to apply a source code formatter, such as indent, on the source code, and then compare the original source code with the formatter’s output. The number of differences found is an indication of how closely the code follows the code style conventions: a large number of differences indicates a poor adherence to the style conventions. A problem of this approach is that for some languages, such as C and C++, there are many acceptable style conventions. In these cases, either the code style tool has to be configured according to the documented code conventions, or the code’s conventions have to be deduced from the actual code [31].

The following shell script will deduce the code’s conventions by perturbing the (far too many) settings passed to the indent program, and keeping the setting that minimizes the number of lines that do not match the specified style each time. After it is executed with FILES set to a (hopefully representative) set of files on which to operate, it will set the variable INDENT_OPT to the indent options that match the code’s style more closely.

Running indent on the Windows Research Kernel without any options results in 389,547 violations found among 583,407 lines. After determining the appropriate indent options with the preceding script (-i4 -ts0 -bli0 -c0 -cd0 -di0 -bad -bbb -br -brs -bfda -bfde -nbbo -ncs) the number of lines found to be violating the style conventions shrinks to 118,173. This type of analysis can pinpoint, for example, developers and teams that require additional mentoring or training to help them adhere to an organization’s code style standards. An increase of these figures over time can be an indicator of stress in an organization’s development processes.

7.4.1 Assembly Language

Most compilers provide a switch that directs them to produce assembly language source code, rather than binary object code. The corresponding flag for most Unix compilers is -S. Assembly language files can be easily processed using text tools, such as grep, sed, and awk. As an example, we will see a script that counts the code’s basic blocks.

A basic block is a portion of code with exactly one entry and exit point. It can be valuable to analyze code in terms of basic blocks, since it allows measuring things like code complexity and test coverage requirements.

We can obtain information regarding the basic blocks of gcc-compiled code by passing to the compiler the --coverage flag in conjunction with the -S flag to produce assembly language output. The generated code at the entry or exit of a basic block looks like the following excerpt (without the comments).

From the above code it is easy to see that the counter associated with each basic block occupies 8 data bytes. The compiler stores the counters in common data blocks allocated on a per-function basis, like the ones in the following example obtained by analyzing a small C program.⁷

The three arguments to each lcomm pseudo-op are the block’s name, its size, and alignment. By dividing the size by 8 we can obtain the number of basic block boundaries associated with each function. Thus, we can process a set of assembly language files produced by compiling code with the --coverage option and then use the following script to obtain a list of functions ordered by the number of basic blocks boundaries embedded in them.

Here is an example of the script’s output for the preceding program.

The number of basic blocks in each function can be used to assess code structure, modularity, and (together with other metrics) locate the potential trouble spots.

7.4.2 Machine Code

On Unix systems we can analyze object files containing machine code with the nm program.⁸ This displays a list of defined and undefined symbols in each object file passed as an argument [33, pp. 363–364]. Defined symbols are preceded by their address, and all symbols are preceded by the type of their linkage. The most interesting symbol types found in user-space programs in a hosted environment are the following.

B Large uninitialized data; typically arrays

C Uninitialized “common”data; typically variables of basic types

D Initialized data

R Read-only symbols (constants and strings)

T Code (known as text)

U Undefined (imported) symbol (function or variable)

Lowercase letter types (e.g., “d”or “t”) correspond to symbols that are defined locally in a given file (with a static declaration in C/C++ programs).

Here is as an example the output of running nm on a C “hello, world” program.

As a first example of this technique, consider the task of finding all symbols that a (presumably) large C program should have declared as static. Appropriate static declarations, minimize name space pollution, increase modularity, and can prevent bugs that might be difficult to locate. Therefore, the number of elements in such a list could be a metric of a project’s maintainability.

Our second example derives identifier metrics according to their type. The script we will see can analyze systems whose object files reside in a directory hierarchy and therefore uses find to locate all object files. After listing the defined symbols with nm, it uses awk to tally in an associative array (map) the count and total length of the identifiers of each identifier category. This allows it in the end to print the average length and count for each category.

Running the preceding script on a compiled kernel of Freebsd produces the following results.

From these results we can see that in each category there are more local (static) symbols (identified by a lowercase letter) than global ones (shown with the corresponding uppercase letter), and that global functions (T) are far more common (12567) than global variables (D: 1649) and arrays (B: 1229). This allows us to reason regarding the encapsulation mechanisms used in the specific system.

Binary code analysis can go into significantly more depth than the static analysis techniques we have discussed in other sections. The code sequences can be analyzed in considerably more detail, while dynamic analysis methods can be used to obtain information from running programs. Tool support can help in both regards; a recently published survey of available tools [34] provides a good starting point.

7.4.3 Dealing with Name Mangling

Some of the techniques covered in this section work well with code written in older languages, such as C and Fortran. However, if we try them on code written in relatively newer ones, such as Ada, C++, and Objective-C, we will get gibberish, as illustrated in the following example.

The reason for this is name mangling: a method C++ compilers use to reconcile linkers that were designed for simpler languages with the requirement of C++ for type-correct linking across separately compiled files. To achieve this feat the compiler adorns each externally visible identifier with characters that specify its precise type.

We can undo this mangling by passing the resulting text through the c++filt tool, which ships with gnu binutils. This will decode each identifier according to the corresponding rules, and provide the full language-dependent type associated with each identifier. For instance, the demangled C++ identifiers of the preceding example are the following.

7.4.4 Byte Code

Programs and libraries associated with the jvm environment are compiled into portable byte code. This can be readily analyzed to avoid the complexity of analyzing source code. The javap program, which comes with the Java development kit, gets as an argument the name of a class, and, by default, prints its public, protected, and package-visible members. For instance, this is the javap output for a “hello, world” Java program.

Here is how we can use the output of javap to extract some basic code metrics of Java programs (in this case the ClassGraph class).

The javap program can also disassemble the Java byte codes contained in each class file. This allows us to perform even more sophisticated processing. The following script prints virtual method invocations, ordered by the number of times each class invokes a method.

Running the above script on the umlgraph’s compiled method allows us to determinate that the org.umlgraph.doclet.Options class calls the method String.equals 158 times. Measures like these can be used to build a dependency graph, which can then expose refactoring opportunities.

If the output of javap is too low-level for our purpose, another alternative for processing Java byte code is the FindBugs program [35]. This allows the development of plug-ins that are invoked when specific code patterns are encountered. For instance, a simple plugin can detect instances of BigDecimal types that are created from a double value,⁹ while a more complex one can locate arbitrary errors in method arguments that can be determined at the time of the analysis [36].

7.4.5 Dynamic Linking

Modern systems have their executable programs link dynamically at runtime to the libraries they require in order to run. This simplifies software updates, reduces the size on disk of each executable program, and allows the running programs to share each library’s code in memory [37, p. 281]. Obtaining a list of the dynamic libraries a program requires allows us to extract information regarding software dependencies and reuse.

On Unix systems the ldd program will provide a list of the libraries an executable program (or an other library) requires in order to run. Here is an example of running the ldd command on bin/ls on a Linux system.

As usual, we can then process this output with additional tools to generate higher-level results. As an example, the following pipeline will list the libraries required by all programs in the /usr/bin directory, ordered by the number of programs that depend on them. The pipeline’s output can be used to study the dependencies between modules and software reuse.

These are the first ten lines of the preceding pipeline’s output on a Freebsd system,

and these on a Linux system.

7.4.6 Libraries

Library files contain compiled object files packed together so that they can be easily shipped and used as a unit. On Unix systems nm can be applied to see the symbols defined and referenced by each library member (see Section 7.4.2), while the ar program can be run to list the files contained in the library. As an example, the following pipeline will list the C library’s files ordered by their size.

Here are the first few lines of the pipeline’s output on a Linux system.

Such a listing can provide insights on a library’s modularity, and modules that could be refactored into units of a more appropriate size.

The corresponding program for Java archives is jar. Given the name of a Java .jar file it will list the class files contained in it. The results can then be used by other programs for further processing.

Consider the task of calculating the Chidamber and Kemerer metrics [38] for a set of classes. These metrics comprise for a class the following following values:

WMC : Weighted methods per class

DIT : Depth of inheritance tree

NOC : Number of children

CBO : Coupling between object classes

RFC : Response for a class

LCOM : Lack of cohesion in methods

The metrics can be used to assess the design of an object-oriented system and to improve the corresponding process.

The following example will calculate the metrics for the classes contained in the ant.jar file and print the results ordered by the weighted methods per class. For the metrics calculation it uses the ckjm program [39],¹⁰ which expects as its input pairs of files and class names.

Here are, as usual, the first few lines of the pipeline’s output.

The preceding metrics of object-oriented code can be further processed to flag classes with values that merit further analysis and justification [37, pp. 341–342], [40], and to locate opportunities for refactoring. In this case, some of the preceding classes, which comprise more than 50 classes each, may need refactoring, because they violate a rule of a thumb stating that elements consisting of more than 30 subelements are likely to be problematic [41, p. 31].

7.5.1 Obtaining Repository Data

Before a repository’s data can be analyzed, it is usually preferable to obtain a local copy of the repository’s data [55]. Although some repositories allow remote access to clients, this access is typically provided to serve the needs of software developers, i.e., an initial download of the project’s source code, followed by regular, but not high-frequency, synchronization requests and commits. In contrast, a repository’s analysis might involve many expensive operations, like thousands of checkouts at successive time points, which can stress the repository server’s network bandwidth, cpu resources, and administrators. Operations on a local repository copy will not tax the remote server, and will also speed up significantly the operations run on it.

The techniques used for obtaining repository data depend on the repository type and the number of repositories that are to be mirrored. Repositories of distributed version control systems [56] offer commands that can readily create a copy of a complete repository from a remote server, namely bzr branch for Bazaar [57], git clone for Git, and hg clone for Mercurial [58]. Projects using the Subversion or cvs system for version control can sometimes be mirrored using the rsync or the svnsync (for Subversion) command and associated protocol. The following are examples of commands used to mirror a diverse set of repositories.

Using svnsync is more involved; here is an example of the commands used for mirroring the Subversion repository of the jboss application server.

Alternatively, if a friendly administrator has shell-level access to the site hosting the Subversion repository, the repository can be dumped into a file and restored back from it using the commands svnadmin dump and svnadmin load.

When multiple repositories are hosted on the same server, their mirroring can be automated by generating cloning commands. These can often be easily created by screen-scrapping a web page that lists the available repositories. As an example consider the Git repositories hosted on git.gnome.org.

Each project is listed with an html line like the following.

The name of the corresponding Git repository can be readily extracted from the url. In addition, because the projects are listed in multiple web pages, a loop is required to iterate over them, specifying the project list’s offset for each page. The following short script will download all repositories using the techniques we saw.

In recent years GitHub has evolved to be an amazingly large repository of open source projects. Although GitHub offers an api to access the corresponding project data (see Figure 7.2), it does not offer a comprehensive catalog of the data stored in it [59]. Thankfully, large swathes of the data can be obtained as database dump files in the form of a torrent [17].

7.5.2 Analyzing Metadata

Metadata analysis can easily be performed by running the version control system command that outputs the revision log. This can then be analyzed using the process outlined in Section 7.2.

As examples of metadata, consider the following Git log entry from the Linux kernel

and an older cvs log entry from Freebsd’s sed program.

The following example illustrates how we can obtain the time of the first commit from various types of repositories.

Some version control systems, such as Git, allow us to specify the format of the resulting log output. This makes it easy to isolate and process specific items. The following sequence will print the author names of the 10 most prolific contributors associated with a Git repository, ordered by the number of commits they made.

The result of running the preceding script on the last 10 years of the Linux kernel is the following.

Such lists can be used to gain insights into the division of labor within teams and developer productivity.

Aggregate results can be readily calculated using awk’s associative arrays. The following example shows the lines contributed by each developer in a cvs repository.

The first 10 lines from the output of the preceding command run on the FreeBSD kernel are the following.

7.5.3 Analyzing Time Series Snapshots

Creating a series of source code snapshots from a repository requires us

• to perform accurate data calculations,

• to represent the dates in a format that can be unambiguously parsed by the repository’s version control system, and

• to check out the corresponding version of the software.

Accurate date calculations on time intervals can be performed by expressing dates in seconds from the start of an epoch (1970-01-01 on Unix systems). The Unix date command allows the conversion of an arbitrary start date into seconds since Epoch. Unfortunately, the way this is done differs between various Unix-like systems. The following Unix shell function expects as its first argument a date expressed in iso-8601 basic date format (YYYYMMDD). It will print the date as an integer representing seconds since Epoch.

The reverse conversion, from Epoch seconds to the iso-8601 extended format (YYYY-MM-DD), which most version control systems can parse unambiguously, again depends on the operating system flavor. The following Unix shell function expects as its first argument a date expressed as seconds since Epoch. It will print the corresponding date in iso format.

As we would expect, the code used to check out a snapshot of the code for a given date depends on the version control system in use. The following Unix shell function expects as its first argument an iso-8601 extended format date. In addition, it expects that the variable $REPO is set to one of the known repository types, and that it is executed within a directory where code from that repository has already been checked out. It will update the directory’s contents with a snapshot of the project stored in the repository for the specified date. In the case of a Bazaar repository, the resulting snapshot will be stored in /tmp/bzr-checkout.

Given the building blocks we saw, a loop to perform some processing on repository snapshots over successive 10 day periods starting from, say, 2005-01-01, can be written as follows.

7.5.4 Analyzing a Checked Out Repository

Given a directory containing a project checked out from a version control repository, we can analyze it using the techniques listed in Section 7.3. Care must be taken to avoid processing the data files associated with the version control system. A regular expression that can match these files, in order to exclude them, can be set as follows, according to the repository type.

Another prerequisite for the analysis is identifying the source code files to analyze. Files associated with specific programming languages can be readily identified by their extension. For instance, C files end in .c; C++ files typically end in .cpp, .C, .cc, or .cxx; while Java files end in .java. Therefore, the following command

will output all the Java source code files residing in the current directory tree.

On the other hand, if we wish to process all source code files (e.g., to count the source code size in terms of lines) we must exclude binary files, such as those containing images, sound, and compiled third-party libraries. This can be done by running the Unix file command on each project file. By convention the output of file will contain the word text only for text files (in our case source code and documentation).

Putting all the above together, here is a pipeline that measures the lines in a repository snapshot checked out in the current directory.

7.5.5 Combining Files with Metadata

The version control system can also be used to help us analyze a project’s files. An invaluable feature is the annotation (also known as “blame”) command offered by many version control systems. This will display a source code file, listing with each line the last commit associated with it, the committer, and the corresponding date.

Given such a list we can easily cut specific columns with the Unix cut command, and analyze version control metadata at the level of source code lines rather than complete files. For example, the following command will list the top contributors in the file cgroup.c of the Linux kernel at its current state.

This is the command’s output.

Similarly, we could find how many lines of the file stem from each year.

This is the output we get by the preceding command.

7.5.6 Assembling Repositories

Projects with a long history provide an interesting source data. However, the data are seldom stored neatly in a single repository. More often than not we find snapshots from the beginning of a project’s lifetime, then one or more frozen dumps of version control systems that are no longer used, and, finally, the live version control system. Fortunately, Git and other modern version control systems offer mechanisms to assemble a project’s history retroactively, by piecing together various parts.

With Git’s graft feature, multiple Git repositories can be pieced together into a whole. This is, for instance, the method Yoann Padioleau used to create a Git repository of Linux’s history, covering the period 1992–2010.¹¹ The last repository in the series is the currently active Linux repository. Therefore, with a single git pull command, the archive can be easily brought up to date. The annotated Linux file in Section 7.5.5 stems from a repository assembled in this way. A similar repository¹² covers 44 years of Unix development history [60].

If the repositories are not in Git format, then, given Git’s flexibility, the most expedient way to combine them into Git format is to use the methods we saw in this section for analyzing modern repositories.

Snapshots of a project can be imported into a Git repository with the correct date and committer (which must be derived from external sources, such as timestamps), using a Unix shell function like the following. This expects as its first argument the directory where the snapshot’s files are located, and as its second argument an iso-8601 basic date format (YYYYMMDD) date associated with the snapshot. When run within a directory where a Git repository has been checked out, it will add the files to the repository with a commit dated as specified. The code utilizes the iso_b_to_epoch function we saw in Section 7.5.3. To specify the code’s author the git commit --author flag can be added to the code.

Importing into Git data stored in other repositories is relatively easy, thanks to existing libraries and tools that can be used for this purpose. Of particular interest is the (badly misnamed) cvs2svn program,¹³ which can convert rcs [61] and cvs repositories into Git ones. In addition, the Perl vcs-sccs library¹⁴ contains an example program that can convert legacy sccs [62] repository data into Git format.

Finally, if a program that can perform the conversion cannot be found, a small script can be written to print revision data in Git’s fast import format. This data can then be fed into the git fast-import command, to import it into Git. The data format is a textual stream, consisting of commands, such as blob, commit, tag, and done, which are complemented by associated data. According to the command’s documentation and the personal experience of this chapter’s author, an import program can be written in a scripting language within a day.

7.6.1 Graphs

Perhaps the most impressive tool of those we will examine is dot. Part of the Graphviz suite [20], originally developed by AT&T, it lets us describe hierarchical relations between elements using a simple declarative language. For instance, with the following input dot will generate the diagram shown in Figure 7.3.

Dot offers a wide choice of node shapes, arrows, and options for controlling the graph’s layout. It can handle graphs with thousands of nodes. This study’s author has used it to display class hierarchies, database schemas, directory trees, package dependency diagrams, mechanical gear connections, and even genealogical trees. Its input language is simple (mostly graph edges and nodes), and it is trivial to generate a diagram from a script of just a few lines.

For example, the following Perl script will create a diagram of a directory tree. When run on the Linux source code tree, it will generate the Figure 7.4[31]. This shows a relatively shallow and balanced tree organization, which could be a mark of an organization that maintains equilibrium between the changes brought by organic growth and the order achieved through regular refactorings. An architect reviewer might also question the few deep and narrow tree branches appearing in the diagram.

It is also easy to create diagrams from the version control system’s metadata. The following Unix shell script will create a diagram of relationships between Linux authors and committers. The result of processing the first 3000 lines of the Linux kernel Git log can be seen in Figure 7.5.

Three cousins of dot, also parts of GraphViz, are neato, for drawing undirected graphs, and twopi and circo, for drawing radial and circular layout graphs. All use an input language similar to dot’s. They are less useful for visualizing software systems, but in some cases they come in handy. For instance, this author has used neato has to draw the relationships between software quality attributes, links between Wikipedia nodes, and collaboration patterns between software developers.

7.6.2 Declarative Diagrams

A slightly less declarative, but no less versatile, family of tools are those that target text-based typesetting systems: TikZ [63], which is based on TE X [64], and pic [65], which is based on troff [66]. The pic program was originally developed at AT&T’s Bell Labs as part of the Unix document preparation tools [67], but these days it is more likely to appear in its GNU groff reincarnation. Pic’s language gives us commands such as box, circle, line, and arrow. Unlike the GraphViz tools, pic will not lay out the diagram for us, but it makes up for its lack of intelligence by letting us create macros and supporting loops and conditionals. This lets us define our own complex shapes (for our project’s specialized notation) and then invoke them with a simple command. In effect, we are creating our own domain-specific drawing language. As an example, the following pic code, in conjunction with macros defined as part of the UMLGraph system [30], will result in Figure 7.6.

7.6.3 Charts

When dealing with numbers, two useful systems for generating charts are gnuplot¹⁵ [68] and the R Project [69]. Gnuplot is a command-line driven graphing utility, whereas R is a vastly larger and more general system for performing statistical analysis, which also happens to have a very powerful plotting library.

Gnuplot can plot data and functions in a wide variety of 2D and 3D styles, using lines, points, boxes, contours, vector fields, surfaces, and error bars. We specify what our chart will look like with commands like plot with points and set xlabel. To plot varying data (e.g., to track the number of new and corrected bugs in a project), we typically create a canned sequence of commands that will read the data from an external file our code generates.

As an example of using gnuplot to draw a chart from software-generated data, consider the task of plotting a program’s stack depth [37, p. 270]. The stack depth at the point of each function’s entry point can be obtained by compiling the program’s code with profiling enabled (by passing the -pg flag to gcc), and using the following custom profiling code to write the stack depth at the point of each call into the file pointed by the file descriptor fd.

Then, a small script, such as the following one written in Perl, can read the file and create the corresponding gnuplot file, which will then create a chart similar to the one seen in Figure 7.7. The information gathered from such a figure can be used to judge the size and variation of the program’s stack size and therefore allow the tuning of stack allocation in memory-restricted embedded systems.

More sophisticated diagrams can be plotted with R and the ggplot2 library [70].

7.6.4 Maps

The last domain we will covere involves geographical data. Consider data like the location of a project’s contributors, or the places where a particular software is used. To place the corresponding numbers on the map, one option is the Generic Mapping Tools (GMT) [21].¹⁶ We use these by plumbing together 33 tools that manipulate data and plot coastlines, grids, histograms, lines, and text using a wide range of map substrates and projections. Although these tools are not as easy to use as the others we have covered, they create high-quality output and offer extreme flexibility in a demanding domain.

As an example, consider the map depicting the contributions of Freebsd developers around the world (Figure 7.8), showing that development is mainly concentrated in Europe, North America, and Japan. The map was generated using the following script, which ends with two gmt commands. Since this is the last script of this work, it brings together many of the techniques we have examined, integrating process and product data with their visualization.

Another alternative involves generating kml, the Google Earth xml-based file format, which we can then readily display through Google Earth and Maps. The limited display options we get are offset by the ease of creating kml files and the resulting display’s interactivity.

If none of the tools we have seen fits our purpose, we can dive into lower-level graphics languages such as PostScript and svg (Scalable Vector Graphics). This approach has been used to annotate program code [33] and to illustrate memory fragmentation [37, p. 251]. Finally, we can always use ImageMagick¹⁷ to automate an image’s low-level manipulation.

The tools described in this section offer a bewildering variety of output formats. Nevertheless, the choice is easy. If we are striving for professional-looking output, we must create vector-based formats such as PostScript, pdf, and svg; we should choose the format our software best supports. The resulting diagrams will use nice-looking fonts and appear crisp, no matter how much we magnify them. On the other hand, bitmap formats, such as png, can be easier to display in a presentation, memo, or web page. Often the best way to get a professional-looking bitmap image is to first generate it in vector form and then rasterize it through Ghostscript or a pdf viewer. Finally, if we want to polish a diagram for a one-off job, a clever route is to generate svg and manipulate it using the Inkscape¹⁸ vector-graphics editor.