We recruited eight developers, all of whom said that they spent at least 20 hours per week writing Visual Basic 6 code at work. We asked them to use Visual Basic .NET and the .NET framework to write a program that would write to and read from text files on disk. Each participant was given two hours to complete the task. While working on the task, they were free to browse the documentation available for any of the classes in the .NET framework. Each participant tackled the task individually in the usability labs on our campus in Redmond, WA. A usability engineer was behind the one-way mirror in the usability lab to operate the video cameras and to take notes. Other than helping the participant recover from any bugs or crashes that came up with the pre-release version of Visual Studio, the usability engineer did not help the participant with any of the tasks.

In contrast to the results from the ATL Server card sort study, the results from this study were very enlightening. The headline result was that not one participant could complete any of the tasks successfully in the two hours given.

This was quite unexpected. We didn’t think every participant would be completely successful, but we also hadn’t expected such a complete bust. Our first reaction was that the participants simply had not satisfied the study recruiting profile and were in fact not experienced programmers at all. But it was clear from the video recording of each session that this was not the case. Each participant was clearly an experienced programmer who was not daunted at the prospect of picking up and using a new programming language and environment.

So what was the reason for the poor success rate? We reviewed the videos from each session in detail soon after the study. We had already reviewed each session to create highlight video clips that we distributed internally around the Visual Studio product team almost immediately after the last session had ended. These highlights showed the difficulties that participants had performing each of these tasks, so we had a few hypotheses about the reason for the difficulties.

The first hypothesis we tested was whether the documentation was to blame. This hypothesis was inspired by watching all of the participants spend most of their time in the usability lab searching and hunting through the documentation, looking for some class or method that would help them write to or read from a file on disk. At the time we ran the study (summer 2002), the documentation did not contain any code snippets for common tasks such as writing to or reading from a file on disk.

For example, the transcript of one participant browsing the documentation for the System.IO namespace is typical:

The associated video clip is very compelling and is the one we used extensively to publicize the results of the study throughout the Visual Studio product team. Along with other critical video clips, it generated huge empathy for the customer on the part of the product team and a massive desire to figure out what was broken and how to fix it.

In another video clip, a participant is struggling with a compiler error:

Since a large proportion of each participant’s time was spent browsing the documentation, many people in the product team latched onto that and believed that the best way to improve the experience would be to modify the documentation. If the documentation could provide clear pointers to sample code and code snippets, developers could copy and paste those snippets into their own code.

However, we wanted to be sure that the lack of code snippets truly was the underlying cause of the problems we observed in the usability lab. Although modifying the documentation at this stage (three-quarters of the way through the project life cycle) would be a less costly fix than modifying the APIs, it would be wasted energy if it turned out to have no relationship to the underlying cause.

In order to be confident in our analysis of the problems, we needed a framework or a language upon which to base our analysis. Without such a framework, we would have difficulties explaining and justifying our analysis to a product team that was already convinced by the video clips that the problem was the documentation. We ended up using the Cognitive Dimensions framework [Green and Petre 1996].

The Cognitive Dimensions framework is an approach to analyzing the usability of programming languages and, more generally, any kind of notation, such as an API. The framework consists of 13 different dimensions, each describing and measuring a specific attribute that contributes to the usability of the notation.

Each dimension is effectively a probe into the design of the notation, measuring one attribute of its usability. Each measurement can be compared to a baseline (for example, the preferences of a set of developers), and the results of the measurements and comparisons can then be used to determine any corrective action that needs to be taken to ensure that the notation maps well to the baseline.

The measurements that result from this kind of analysis aren’t numerical or even necessarily absolute measurements, but are instead more like ratings on a scale, defined by each dimension. Although this may introduce some uncertainty about the accuracy of each measurement, the beauty of the framework overall is that it identifies and provides a vocabulary with which to discuss the detailed attributes of the usability of a notation. Without such a detailed vocabulary, any analysis of the usability of a notation is fraught with difficulty, because the lack of labels and definitions for each of the different attributes of the usability of a notation makes it close to impossible to reach a shared understanding of what important attributes need to be considered.

We made some modifications to the names to make the framework more relevant to API usability [Clarke and Becker 2003]. A short description of each dimension follows:

We were attracted to the framework based on claims about its ease of use. Green and Petre claim that the Cognitive Dimensions framework can be used by people without any formal training or background in cognitive psychology [Green and Petre 1996]. We believed that it was vital for the development team reviewing the results of our study to view those results in the context of a framework or language that made sense to them.

We used the framework to help describe the experience that participants had in each task. We reviewed the video recordings of each session and made video clips for each interesting event we observed. We then tagged each event with the set of specific cognitive dimensions that we believed were relevant, and reviewed the video clips in the context of the dimensions associated with them. Thus, we reviewed all the abstraction level clips together, all of the viscosity clips together, etc. This approach gave us a different perspective on the video recordings of each session. We used that perspective to develop and articulate a clearer explanation for the difficulties observed.

Crucially, we were able to use the Cognitive Dimensions framework to convince the product team that fixing the documentation alone would not suffice. Although it was necessary, our analysis told us it would not be sufficient.

The Cognitive Dimensions analysis suggested that the critical underlying problem was related to the abstraction level of the APIs used. To explain this explanation in more detail, let’s consider the code snippet that a participant had to write in order to read code from a text file:

Imports System
Imports System.IO

Class Test
    Public Shared Sub Main()
        Try
            ' Create an instance of StreamReader to read from a file.
            Dim sr As StreamReader = New StreamReader("TestFile.txt")
            Dim line As String
            ' Read and display the lines from the file until the end
            ' of the file is reached.
            Do
                line = sr.ReadLine()
                Console.WriteLine(Line)
            Loop Until line Is Nothing
            sr.Close()
        Catch E As FileNotFoundException
            ' Let the user know what went wrong.
            Console.WriteLine("The file could not be found:")
            Console.WriteLine(E.Message)
        End Try
    End Sub
End Class

Writing to a file is very similar:

Imports System
Imports System.IO

Class Test
    Public Shared Sub Main()
        ' Create an instance of StreamWriter to write text to a file.
        Dim sw As StreamWriter = New StreamWriter("TestFile.txt")
        ' Add some text to the file.
        sw.Write("This is the ")
        sw.WriteLine("header for the file.")
        sw.WriteLine("-------------------")
        ' Arbitrary objects can also be written to the file.
        sw.Write("The date is: ")
        sw.WriteLine(DateTime.Now)
        sw.Close()
    End Sub
End Class

This code doesn’t appear particularly complex. The StreamReader or StreamWriter constructor takes a string representing a path to a file. Then, the reader or writer is used to read from or write to the disk accordingly.

If you already know the solution to these tasks, and you watch the videos of participants struggling to write this code, it is natural to leap to the conclusion that the problem lies with the documentation not showing a code snippet for this task. After all, how could this code be made any simpler?

However, when we analyzed the video clips and the associated talking commentary provided by each participant, we knew that the problem is not the lack of code snippets. For example, reread a portion of the transcript that I quoted previously:

When the participant says that this seems too low-level, he is referring to the abstraction level of the classes he is browsing. Instead of a very flexible, but somewhat abstract, stream-based implementation, participants expected something less abstract and more concrete. The representation of the task in the API was far removed from the way participants expected the task to be represented. Instead, participants had to change the way they thought about text files in order to recognize that the StreamWriter and StreamReader classes were the correct classes to use.

We made this same observation among the majority of the participants. Even when they were browsing the documentation for the class that would allow them to complete the task (StreamReader or StreamWriter) they would dismiss it, because it didn’t map to the way they expected reading and writing to work.

So the problem wasn’t so much that the documentation lacked appropriate code snippets. It was more that participants were searching for something in the documentation that didn’t exist. They expected to find a concrete representation of a text file, but found only representations of file streams.

The Cognitive Dimensions framework gave us a language with which to analyze and better understand the problems we observed in the study. With this analysis, we convinced the team that the best way to fix the problem, in addition to addressing the need for better documentation, would be to design and create a new class that concretely represented the filesystem and files on disk.

Six months later, the team had designed and implemented a prototype version of this new class. The code required to complete the same tasks using this new class (FileObject) follows:

Dim f As New FileObject
f.Open(OpenMode.Write, "testfile.txt")
f.WriteLine("A line of text")
f.Close()

In this case, a file is represented by an instance of the FileObject class. The FileObject class can exist in different modes (in this case, OpenMode.Write opens the file for writing). This contrasts starkly with the earlier code examples that use different classes to represent the different modes (StreamWriter and StreamReader).

We ran a study like our previous one on the new FileObject API, using the same tasks and the same profile of participants (although not the same participants). This time the results were markedly different. Within 20 minutes and without browsing any documentation, all participants were able to complete each task.