6.1 Introduction

Whether they consist of code, bug/issue reports, mailing-list messages, requirements specifications, or documentation, software repositories include text documents. This textual information is an invaluable source of information, and can potentially be used in a variety of software-engineering activities. The textual descriptions of bugs can be compared against each other to recognize duplicate bug reports. The textual documentations of software modules can be used to recommend relevant source-code fragments. E-mail messages can be analyzed to better understand the skills and roles of software developers, and to recognize their concerns about the project status and their sentiments about their work and teammates. This is why we have recently witnessed numerous research projects investigating the application of text-analysis methods to software text assets.

The simplest approach to analyzing textual documents is to use a vector-space model, which views documents (and queries) as frequency vectors of words. For example “the” occurred once, “my” occurred twice, “bagel” occurred zero times, and so on. Effectively, a vector-space model views terms as dimensions in a high-dimensional space, so that each document is represented by a point in that space on the basis of the frequency of terms it includes. This model suffers from two major shortcomings. First, it makes the consideration of all words impractical: since each word is a dimension, considering all words would imply expensive computations in a very high-dimensional space. Second, it assumes that all words are independent. In response to these two assumptions, methods for extracting topic models—that is, thematic topics corresponding to related bags of words, were developed.

A thematic topic is a collection of words which are somehow related. For example, a topic might consist of the words “inning,” “home,” “batter,” “strike,” “catcher,” “foul,” and “pitcher,” which are all related to the game of baseball. The most well known topic-model methods are latent semantic indexing (LSI) and latent Dirichlet allocation (LDA). LSI employs singular-value decomposition to describe relationships between terms and concepts as a matrix. LDA arranges and rearranges words into buckets, which represent topics, until it estimates that it has found the most likely arrangement. In the end, having identified the topics relevant to a document collection (as sets of related words) LDA associates each document in the subject collection with a weighted list of topics.

LDA has recently emerged as the method of choice for working with large collections of text documents. There is a wealth of publications reporting its applications in a variety of text-analysis tasks in general and software engineering in particular. LDA can be used to summarize, cluster, link, and preprocess large collections of data because it produces a weighted list of topics for every document in a collection dependent on the properties of the whole. These lists can then be compared, counted, clustered, paired, or fed into more advanced algorithms. Furthermore, each topic comprises a weighted list of words which can be used in summaries.

This chapter provides an overview of LDA and its relevance to analyzing textual software-engineering data. First, in Section 6.2, we discuss the mathematical model underlying LDA. In Section 6.3, we present a tutorial on how to use state-of-the-art software tools to generate an LDA model of a software-engineering corpus. In Section 6.4, we discuss some typical pitfalls encountered when using LDA. In Section 6.5, we review the mining-software repositories literature for example applications of LDA. Finally, in Section 6.6, we conclude with a summary of the important points one must be aware of when considering using this method.

6.2 Applications of LDA in Software Analysis

The LDA method was originally formulated by Blei et al. [1], and it soon became quite popular within the software-engineering community. LDA’s popularity comes from the variety of its potential applications.

LDA excels at feature reduction, and can employed as a preprocessing step for other models, such as machine learning algorithms. LDA can also be used to augment the inputs to machine learning and clustering algorithms by producing additional features from documents. One example of this type of LDA usage was described by Wang and Wong [2], who employed it in a recommender system. Similarly, labeled LDA can be used to create vectors of independent features from arbitrary feature sets such as tags.

An important use of LDA is for linking software artifacts. There are many instances of such artifact-linking applications, such as measuring coupling between code modules [3] and matching code modules with natural-language requirement specifications [4] for traceability purposes. Asuncion et al. [5] applied LDA on textual documentation and source-code modules and used the topic-document matrix to indicate traceability between the two. Thomas et al. [6] focused on the use of LDA on yet another traceability problem, linking e-mail messages to source-code modules. Gethers et al. [7] investigated the effectiveness of LDA for traceability-link recovery. They combined information retrieval techniques, including the Jenson-Shannon model, the vector space model, and the relational topic model using LDA. They concluded that each technique had its positives and negatives, yet the integration of the methods tended to produce the best results. Typically steeped in the information retrieval domain, Savage et al. [8], Poshyvanyk [9], and McMillan et al. [10] have explored the use of information retrieval techniques such as LSI [11] and LDA to recover software traceability links in source code and other documents. For a general literature survey related to traceability techniques (including LDA), the interested reader should refer to De Lucia et al. [12].

Baldi et al. [13] labeled LDA-extracted topics and compared them with aspects in software development. Baldi et al. claim that some topics do map to aspects such as security, logging, and cross-cutting concerns, which was somewhat corroborated by Hindle et al. [14].

Clustering is frequently used to compare and identify (dis)similar documents and code, or to quantify the overlap between two sets of documents. Clustering algorithms can potentially be applied to topic probability vectors produced by LDA. LDA has been used in a clustering context, for issue report querying, and for deduplication. Lukins et al. [15] applied LDA topic analysis to issue reports, leveraging LDA inference to infer if queries, topics, and issue reports were related to each other. Alipour et al. [16] leveraged LDA topics to add context to deduplicate issue reports, and found that LDA topics added useful contextual information to issue/bug deduplication. Campbell et al. [17] used LDA to examine the coverage of popular project documentation by applying LDA to two collections of documents at once: user questions and project documentation. This was done by clustering and comparing LDA output data.

Often LDA is used to summarize the contents of large datasets. This is done by manually or automatically labeling the most popular topics produced by unlabeled LDA. Labeled LDA can be used to track specific features over time—for example, to measure the fragmentation of a software ecosystem as in Han et al. [18].

Even though LDA topics are assumed to be implicit and not observable, there is substantial work on assessing the interpretability of those summaries by developers. Labeling software artifacts using LDA was investigated by De Lucia et al. [19]. By using multiple information retrieval approaches such as LDA and LSI, they labeled and summarized source code and compared it against human-generated labels. Hindle et al. [20] investigated if developers could interpret and label LDA topics. They reported limited success, with 50% being successfully labeled by the developers, and that nonexperts tend to do poorly at labeling topics on systems they have not dealt with.

Finally, there has been some research on the appropriate choice of LDA hyperparameters and parameters: α, β, K topics. Grant and Cordy [21] were concerned about K, where K is the number of topics. Panichella et al. [22] proposed LDA-GA, a genetic algorithm approach to searching for appropriate LDA hyperparameters and parameters. LDA-GA needs an evaluation measure, and thus Panichella et al. used software-engineering-specific tasks that allowed their genetic algorithm optimized the number of topics for cost-effectiveness.

6.3 How LDA Works

The input of LDA is a collection of documents and a few parameters. The output is a probabilistic model describing (a) how much words belong to topics and (b) how associated topics are with documents. A list of topics, often containing some topics repeatedly, is generated at random on the basis of (b). That list is the same length as the number of words in the document. Then, that list of topics is transformed into a list of words by turning each topic into a word on the basis of (a).

LDA is a generative model. This means that it works with the probability of observing a particular dataset given some assumptions about how that dataset was created. At its core is the assumption that a document is generated by a small number of “topics.” An LDA “topic” is a probability distribution, assigning to each word in the collection vocabulary a probability.

Topics are considered hypothetical and unobservable, which is to say that they do not actually exist in documents. This means that, first, we know that documents are not actually generated from a set of topics. Instead, we are using the concept of a topic as a simplified model for what must be a more complicated process, the process of writing a document. Second, documents do not come with information about what topics are present, what words those topics contain, and how much of each topic is in each document. Therefore, we must infer the topic characteristics from a collection of collections of words. Each document is usually assumed to be generated by a few of the total number of possible topics. So, every word in every document is assumed to be attributable to one of the document’s topics.

Though, of course, words do a have a particular order in a document, LDA does not consider their order. Each word is assigned an individual probability of being generated. That is, the probability of a topic k generating a word v is a value ϕ_k,v [23]. The sum of these probabilities for a topic k must be 1:

$\sum _v{\varphi }_{k,v}=1.$

Furthermore, the ϕ_k,v values are assumed to come from a random variable ϕ_k, with a symmetric Dirichlet distribution (which is the origin of the name of the LDA method). The symmetric Dirichlet distribution has one parameter, β, which determines whether a topic is narrow (i.e., focuses on a few words) or broad (i.e., covers a bigger spread of words). If β is 1, the probability of a topic generating a word often is the same as the probability of a topic generating a word rarely. If β is less than 1, most words will be extremely unlikely, while a few will make up the majority of the words generated. In other words, larger values of β lead to broad topics, and smaller values of β lead to narrow topics.

In summary, words are assumed to come from topics, with the probability of a word coming from a specific topic coming from a Dirichlet distribution. Thus, if we know that β is a very small positive integer, we know that a topic which would generate any word with equal probability is itself very unlikely to exist.

The “document” is an important concept involved in understanding how LDA works. A document is also a probability distribution. Every possible topic has a probability of occurring from 0 to 1, and the sum of these probabilities is 1. The probability that a document d will generate a word from a specific topic k is θ_d,k. Again, the θ_d,k probabilities are probabilities, but the probabilities of θ_d,k taking on a particular probability value comes from a Dirichlet distribution of parameter α. The topics that a document is observed to generate are a vector, Z_d. This vector is N_d words long, representing every word in the document. If α is near 1, we expect to see documents with few topics and documents with many topics in equal proportion. If α is less than 1, we expect most documents to only use a few topics. If α is greater than 1, we expect most documents to use almost every topic.

To summarize, words come from topics. The probability of a word being generated by a specific topic comes from a symmetric Dirichlet distribution. The probability of a document containing a word from a specific topic is dictated by a different symmetric Dirichlet distribution. The words that a document is observed to generate are a vector, W_d, which is formed by observing the topic indicated by the entries in the Z_d vector.

Figure 6.1 shows the words present in each document coming from topic 1 in nine different LDA models of varying parameters. Additionally, it shows α, β, θ, ϕ, Z, and W. By following the arrows, we can see how each prior generates each observed posterior. Figure 6.1 shows the effects that α has on θ, that θ has on Z, and that Z has on W. As we increase α, we observe that more documents contain words from topic 1. Additionally, it shows the effect that β has on ϕ and that ϕ has on W. As we increase β, we end up with topic 1 including a larger variety of vocabulary words. For example, the plot in column 1 and row 1 in Figure 6.1 shows that each document uses a much smaller subset of the possible vocabulary than the plot in column 1 and row 3 below it. Similarly, the plot in column 3 and row 1 shows that many more documents contain words from topic 1 than the plot in column 1 and row 1.

The LDA process consists in allocating and reallocating weights (or probabilities if normalized) in θ and ϕ until the lower bound of the total probability of observing the input documents is maximized. Conceptually, this is accomplished by dividing up topics among words and by dividing up documents among topics. This iterative process can be implemented in many different ways.

Technically, the generative process LDA assumes is as follows, given a corpus of M documents, each of length N_i [1]:

1. For every topic $k\in \left\{1,\dots ,K\right\}$, choose $\overrightarrow{{\varphi }_k}\sim \text{Dir}\left(\beta \right)$.

2. For every document $d\in \left\{1,\dots ,M\right\}$

(a) choose $\overrightarrow{{\theta }_d}\sim \text{Dir}\left(\alpha \right)$.

(b) for every word $j\in 1,\dots ,N_d$ in document d

i. choose a topic $z_{d,j}\sim \text{multinomial}\left(\overrightarrow{{\theta }_d}\right)$.

ii. choose a word $w_{d,j}\sim \text{multinomial}\left(\overrightarrow{{\varphi }_{z_{d,j}}}\right)$.

Thus, the probability of a topic k generating a word v at a position j in a document d is $p\left(w_{d,j}=v\mid \alpha ,\beta ,K\right)$:

${\int }_{\mathrm{\Theta }}\sum _{k=1}^K{\int }_{\mathrm{\Phi }}p\left(w_{d,j}=v\mid \overrightarrow{{\varphi }_k}\right)p\left(z_{d,j}=k\mid \overrightarrow{{\theta }_d}\right)p\left(\overrightarrow{{\varphi }_k}\mid \beta \right)p\left(\overrightarrow{{\theta }_d}\mid \alpha \right)\mathrm{d}\overrightarrow{{\varphi }_k}\mathrm{d}\overrightarrow{{\theta }_d},$

integrating over all possible probability vectors of length K (Θ) and of length V (Φ). The goal of LDA software is to maximize the probability

$p\left(\theta ,Z\mid W,\alpha ,\beta ,K\right)$

by choosing θ and Z given a corpus W and parameters α and β. Unfortunately, this problem is intractable [1], so the values of θ and ϕ that maximize the above probability are estimated by LDA software. The exact technique employed to estimate the maximum differs between different pieces of software.

6.4 LDA Tutorial

In this tutorial, we illustrate the LDA method in the context of analyzing textual data extracted from the issue-tracking system of a popular project.

1. The first task involves acquiring the issue-tracker data and representing it in a convenient format such as JavaScript Object Notation (JSON).

2. Then we transform the text of the input data—namely, we convert the text to word counts, where words are represented as integer IDs.

3. We apply LDA software on the transformed documents to produce a topic-document matrix and a topic-word matrix.

4. We then summarize the top words from the topic-word matrix to produce topic-word summaries, and store the topic-document matrix.

5. Finally, we analyze the document matrix and the topics. The objective of this analysis step is to (a) examine the latent topics discovered, (b) plot the topic relevance over time, and (c) cluster the issues (i.e., input documents) according to their associated topics.

6.4.5 LDA Output Summarization

Our program lda.py produces summary.json, a JSON summary of the top topic words for each topic extracted, ranked by weight. Two other JSON files are created, document_topic_matrix.json and document_topic_map.json. The first file (matrix) contains the documents and weights represented by JSON lists. The second file (map) contains the documents represented by their ID mapped to a list of weights. document_topic_map.json contains both the original ID and the document weight, where as the matrix it uses indices as IDs. lda-topics.json is also produced, and it lists the weights of words associated with each topic, as lists of lists. lids.json is a list of document IDs in the order presented to Vowpal Wabbit and the order used in the document_topic_matrix.json file. dicts.json maps words to their integer IDs. You can download the JSON and comma separated value (CSV) output of our particular run from https://archive.org/29/items/LDAinSETutorial/bootstrap-output.zip.

6.4.5.1 Document and topic analysis

Since the topics have been extracted, let us take a look! In Table 6.1 we see a depiction of 20 topics extracted from the Bootstrap project issue tracker. The words shown are the top 10 ranking words from each of the topics, the most heavily allocated words in the topic.

Table 6.1

The Top 10 Ranked Words of the 20 Topics Extracted from Bootstrap’s Issue-Tracker Issues

Topic No.	Top 10 Topic Words
1	grey blazer cmd clipboard packagist webview kizer ytimg vi wrench
2	lodash angular betelgeuse ree redirects codeload yamlish prototypejs deselect manufacturer
3	color border background webkit image gradient white default rgba variables
4	asp contrast andyl runat hyperlink consolidating negatory pygments teuthology ftbastler
5	navbar class col css width table nav screen http span
6	phantomjs enforcefocus jshintrc linting focusin network chcp phantom humans kevinknelson
7	segmented typical dlabel signin blockquotes spotted hyphens tax jekyllrb hiccups
8	modal input button form btn data http tooltip popover element
9	dropdown issue chrome menu github https http firefox png browser
10	zepto swipe floor chevy flipped threshold enhanced completeness identified cpu
11	grid width row container columns fluid column min media responsive
12	div class li href carousel ul data tab id tabs
13	parent accordion heading gruntfile validator ad mapped errorclass validclass collapseone
14	bootstrap github https css http js twitter docs pull don
15	left rtl support direction location hash dir ltr languages offcanvas
16	percentage el mistake smile spelling plnkr portuguese lokesh boew ascii
17	font icon sm lg size xs md glyphicons icons glyphicon
18	tgz cdn bootstrapcdn composer netdna libs yml host wamp cdnjs
19	npm js npmjs lib http install bin error ruby node
20	license org mit apache copyright xl cc spec gpl holder

Each topic is assigned a number by LDA software; however, the order in which it assigns numbers is arbitrary and has no meaning. If you ran LDA again with different seeds or a different timing (depending on the implementation), you would get different topics or similar topics but in different orders. Nonetheless, we can see in Table 6.1 that many of these topics are related to the Bootstrap project. Topic summaries such as these are often your first canary in the coal mine: they give you some idea of the health of your LDA output. If they are full of random tokens and numbers, one might consider stripping out such tokens from the analysis. If we look to topic 20, we see a set of terms: license org mit apache copyright xl cc spec gpl holder. MIT, Apache, GPL, and CC are all copyright licenses, and all of these licenses have terms and require attribution. Perhaps documents related to topic 20 are related to licensing. How do we verify if topic 20 is about licensing or not?

Using the document-topic matrix, we can look at the documents that are ranked high for topic 20. Thus, we can load the CSV file, document_topic_map.csv, or the JSON file, document_topic_map.json, with our favorite spreadsheet program (LibreOffice is included with the virtual machine), R, or Python, and sort then data in descending order on the T20 (topic 20) column. Right at the top is issue 2054. Browsing large.json or by visiting issue 2054 on Github,⁵ we can see that the subject of the issue is “Migrate to MIT License.” The next issues relate to licensing for image assets (#3942), JavaScript minification (unrelated, but still weighted heavily toward topic 20) (#3057), phantomJS error (#10811), and two licensing issues (#6342 and #966). Table 6.2 provides more details about these six issues. The LDA Python program also produces the file document_topic_map_norm.csv, which has normalized the topic weights. Reviewing the top weighted documents from the normalized CSV file reveals different issues, but four of the six top issues are still licensing relevant (#11785, #216, #855, and #10693 are licensing related but #9987 and #12366 are not). Table 6.3 provides more details about these six normalized issues.

Table 6.2

Issue Texts of the Top Documents Related to Topic 20 (Licensing) from document_topic_map.csv

https://github.com/twbs/bootstrap/issues/2054	cweagans
Migrate to MIT License
I’m wanting to include Bootstrap in a Drupal distribution that I’m working on. Because I’m using the Drupal.org packaging system, I cannot include Bootstrap because the APLv2 is not compatible with GPLv2 …
https://github.com/twbs/bootstrap/issues/3942	justinshepard
License for Glyphicons is unclear
The license terms for Glyphicons when used with Bootstrap needs to be clarified. For example, including a link to Glyphicons on every page in a prominent location isn’t possible or appropriate for some projects. …
https://github.com/twbs/bootstrap/issues/3057	englishextra
bootstrap-dropdown.js clearMenus() needs ; at the end
bootstrap-dropdown.js when minified with JSMin::minify produces error in Firefox error console saying clearMenus()needs ; …
https://github.com/twbs/bootstrap/issues/10811	picomancer
“PhantomJS must be installed locally” error running qunit:files task
I’m attempting to install Bootstrap in an LXC virtual machine, getting “PhantomJS must be installed locally” error. …
https://github.com/twbs/bootstrap/issues/6342	mdo
WIP: Bootstrap 3
While our last major version bump (2.0) was a complete rewrite of the docs, CSS, and JavaScript, the move to 3.0 is equally ambitious, but for a different reason: Bootstrap 3 will be mobile-first. …
MIT License is discussed.
https://github.com/twbs/bootstrap/issues/966	andrijas
Icons as font instead of img
Hi Any reason you opted to include image based icons in bootstrap which are limited to the 16px dimensions? For example http://somerandomdude.com/work/iconic/ is available as open source fonts—means you can include icons in headers, buttons of various size etc since its vector based. …
License of icons is discussed.

Table 6.3

Issue Texts of the Top Normalized Documents Related to Topic 20 (Licensing) from document_topic_map_norm.csv

https://github.com/twbs/bootstrap/pull/12366	mdo
Change a word
…
Blank + Documentation change
https://github.com/twbs/bootstrap/pull/9987	cvrebert
Change ‘else if‘ to ‘else‘
…
Blank + Provided a patch changing else if to else
https://github.com/twbs/bootstrap/pull/10693	mdo
Include a copy of the CC-BY 3.0 License that the docs are under
This adds a copy of the Creative Commons Attribution 3.0 Unported license to the repo. /cc @mdo
https://github.com/twbs/bootstrap/issues/855	mistergiri
Can i use bootstrap in my premium theme?
Can i use bootstrap in my premium cms theme and sell it?
https://github.com/twbs/bootstrap/issues/216	caniszczyk
Add CC BY license to documentation
At the moment, there’s no license associated with the bootstrap documentation. We should license it under CC BY as it’s as liberal as the software license (CC BY). …
https://github.com/twbs/bootstrap/issues/11785	tlindig
License in the README.md
At bottom of README.md is written: Copyright and license Copyright 2013 Twitter, Inc under the Apache 2.0 license. With 3.1 you switched to MIT. It looks like you forgott to update this part too.

6.4.5.2 Visualization

Looking at the numbers and topics is not enough, usually we want to visually explore the data to tease out interesting information. One can use simple tools such as spreadsheets to make basic visualizations.

Common visualization tasks with LDA include the following:

• Plotting document to topic association over time.

• Plotting the document-topic matrix.

• Plotting the document-word matrix.

• Plotting the association between two distinct kinds of documents within the same LDA run.

Given the CSV files, one can visualize the prevalence of topics over time. Figure 6.2 depicts the proportional topic weights of the first 128 issues over time against topics 15-20 from Table 6.1.

From the spreadsheet inspection, the reader should notice that the document topic weights are somewhat noisy and hard to immediately interpret. For instance, it is hard to tell when a topic is popular and when it becomes less popular. Alternatively, one might ask if a topic is constantly referenced over time or if it is periodically popular. One method of gaining an overview is to bin or group documents by their date (e.g., weekly, biweekly, monthly) and then plot the mean topic weight of one topic per time bin over time. This allows one to produce a visualization depicting peaks of topic relevance over time. With the tutorial files we have included an R script called plotter.R that produces a summary of the 20 topics extracted combined with the dates extracted from the issue tracker. This R script produces Figure 6.3, a plot of the average relevance of documents per 2-week period over time. This plot is very similar to the plots in Hindle et al. [20]. If one looks at the bottom right corner of Figure 6.3, in the plot of topic 20 one can see that topic 20 peaks from time to time, but is not constantly discussed. This matches our perception of the licensing discussions found within the issue tracker: they occur when licenses need to be clarified or change, but they do not change all the time. This kind of overview can be integrated into project dashboards to give managers an overview of issue-tracker discussions over time.

Further directions for readers to explore include using different kinds of documents, such as documentation, commits, issues, and source code, and then relying on LDA’s document-topic matrix to link these artifacts. We hope this tutorial has helped illustrate how LDA can be used to gain an overview of unstructured data within a repository and infer relationships between documents.

6.5 Pitfalls and Threats to Validity

This section summarizes the threats to validity that practitioners may face when using LDA. In addition, this section describes potential pitfalls and hazards in using LDA.

One pitfall is that different pieces of LDA software output different types of data. Some LDA software packages report probabilities, while others report word counts or other weights. While one can convert between probabilities and word counts, it is important to consider whether each document receives equal weight, or whether longer documents should receive more weight than shorter documents.

6.6 Conclusions

LDA is a powerful tool for working with collections of structured, unstructured, and semistructured text documents, of which there are plenty in software repositories. Our literature review has documented the abundance of LDA applications in software analysis, from document clustering for issue/bug de-duplication, to linking for traceability between code, documentation, requirements, and communications, to summarizing association of events and documents with software life cycle activities.

We have demonstrated a simple case of using LDA to explore the contents of an issue-tracker repository and showed how the topics link back to documents. We also discussed how to visualize the output.

LDA, however, relies on a complex underlying probabilistic model and a number of assumptions. Therefore, even though off-the-shelf software is available to compute LDA models, the user of this software must be aware of potential pitfalls and caveats. This chapter has outlined the basics of the underlying conceptual model and discussed these pitfalls in order to enable the informed use of this powerful method.