Introduction

With the popularity and availability of Open Source Software (OSS) projects, Software Engineering (SE) researchers have made many advances in understanding how software is developed. However, in SE Research, like in any other scientific field, it is always desirable to produce results, techniques, and tools that can apply to a large (or all if possible) number of software projects. The ideal case would be to randomly select a statistically significant sample of software projects. However, past SE studies evaluate hypotheses on a small sample of deliberately chosen OSS projects that are out there in the world. More recently, an increasing number of SE researchers have started examining their hypotheses on larger datasets, which are deliberately chosen as well. The aim of the large-scale studies is to increase the generality of the research studies. However, generality of results may not be achieved if the sample of projects chosen for evaluation are homogeneous in nature and not diverse with respect to the entire population of SE projects.

To better understand the challenge of sampling, consider a researcher who wants to study distributed development in a large number of projects in an effort to increase generality. Now, consider two possible strategies to get the sample of projects:

• The researcher goes to json.org and selects 20 projects (all of the JSON parsers) that cover a wide variety of programming languages (Java, C#, Python, Ruby, Perl, etc.). Then, any findings will not be representative because of the narrow range of functionality of the projects in the sample.

• The researcher goes to gnu.org and selects 20 projects (all of them written in C) that cover a wide range of domains (eg, compilers, DBMS, editors, games, web browsers, etc.). Then, any findings will not be representative because of the narrow range of languages of the projects in the sample.

These two situations are extreme cases, but help illustrate the importance of systematically selecting the sample of appropriate projects for an empirical study. In this chapter, we present the initial work done on diversity and representativeness in SE research. We first define what we mean by diversity and representativeness in SE research. Then, we present: (a) a way to assess the quality of a given sample of projects with respect to diversity and representativeness and (b) a selection technique that allows one to tailor a sample with high diversity and representativeness.

What Is Diversity and Representativeness?

Diversity and representativeness are two complementary aspects of the quality of a sample selected from a population. A diverse sample contains members of every subgroup in the population, and within the sample, the subgroups have roughly equal size. In a representative sample, the size of each subgroup in the sample is proportional to the size of that subgroup in the population. For example, if the population is comprised of two subgroups: 400 members of type X and 100 members of type Y, then 25X and 25Y would be considered a diverse sample, and 40X and 10Y a representative sample. Following are definitions of the terms needed to formally define diversity and representativeness.

• A large set of projects in a domain are defined as the universe. Examples of SE universes are: all open-source projects, all web applications, all mobile phone applications, all Java applications, etc.

• Each project within a universe is characterized by one or more dimensions. Examples of dimensions are: size in lines of code (LOC), main programming language, number of developers, rating, price, etc. The subset of dimensions that are relevant for research topics are referred to as the space of the research topic.

• For each dimension d in the space of the research topic, a similarity function is defined: $\mathrm{similar}d\left(p1,p2\right)\to \left\{0,1\right\}$ that decides whether projects p1 and p2 are similar at dimension d. The collection of similarity functions in the space are referred to as the configuration of the space. Now, two projects p1 and p2 are similar, if all their dimensions are similar or $\mathrm{similar}\left(p1,p2\right)={\prod }_d\mathrm{similar}d\left(p1,p2\right)$.

• The subgroup of projects (within the universe) that are similar to one another, are in the same neighborhood.

Based on the preceding definitions, the sample coverage (the numerical representation of the amount of the universe covered by the given sample of projects) of the set of projects P for a given universe of projects U is computed as follows: $\mathrm{coverage}\left(P\right)=\left|U_{p\in P}\left\{q|\mathrm{similar}\left(p,q\right)\right\}\right|/\left|U\right|$.

Depending on the research studies, the universe, space, and similarity function can vary. In fact, it is up to the researcher to define the most suitable similarity function for their research topic (space) and target population (universe). In addition, it is also important to discuss the context in which the coverage was computed. The researcher should always keep in mind these questions: What projects is the research intending to be relevant for (universe)? What criteria matter for findings to hold for other projects (space, configuration)?

What Can We Do About It?

Now that we know how to assess the coverage of a given sample, the next step is to learn how to systematically tailor a sample with maximum coverage. The selection technique is a hybrid strategy that combines ideas from Diversity and Representativeness. The main parts of the technique can be summarized as follows:

• Taking ideas from diversity, identify the neighborhoods (subgroups) of all projects in the population. Here, it is important to note that: if two projects are similar, their neighborhoods will overlap.

• Taking ideas from representativeness, select the project based on the size of their neighborhood not yet covered by the sample. With the “not yet covered” condition, the projects in the sample are ensured to not share neighborhoods.

• For selecting a sample of K projects with high coverage, the previous step is repeated, at most, K times.

A detailed description of the algorithms to compute the sample-selection and the sample-coverage can be found in prior work [1]. Additionally, an R implementation of the technique is provided in the online repository. This way, the interested reader can easily use them either in her/his research or to complement the reading of the present chapter.

Recommendations

So far, based on the results from past work, the reader may be tempted to think that studies with low coverage do not contribute much to the body of knowledge in SE. On the contrary, we think that coverage scores do not increase or decrease the importance of research, but rather, enhance our ability to reason about it and understand the context under which the results are applicable.

In SE research it is a common practice to have a section that summarizes the characteristics of the studied projects. We think that this section is the appropriate place to report the coverage of such projects and discuss the target population to be researched (universe) as well as the dimensions relevant for the research (space).

Future Work

In the future we need to examine how and when the preceding approach does not actually generalize results. We also need to examine if choosing projects randomly is better than choosing projects that cover a certain neighborhood of projects in the universe, or a possible hybrid between the approach described herein and random sampling.