As a strong advocate of evidence-based software engineering ([Dybå et al. 2005], [Kitchenham et al. 2004]), I am also a strong advocate of systematic reviews (SRs) ([Kitchenham 2004], [Kitchenham and Charters 2007]). These are sometimes referred to as systematic literature reviews in software engineering to avoid confusions with inspection methods (i.e., methods for reading and reviewing software engineering documents or code). We cannot have evidence-based software engineering without a sound methodology for aggregating evidence from different empirical studies. SRs provide that methodology.

SRs have been in widespread use in other disciplines for decades. Each SR is launched by a researcher to investigate all available evidence that supports or refutes a particular “topic of interest,” which in software engineering typically involves asking about the effect of a method or process. A researcher conducting an SR selects empirical studies that are relevant to the particular research question, assesses the validity of each one, and then determines the trend shown by those studies. Thus, SRs aim to find, assess, and aggregate all relevant evidence about some topic of interest in a fair, repeatable, and auditable manner.

This chapter introduces the value of SRs to readers with a general interest in empirical software engineering. I also aim to help novice researchers (such as PhD students)—who might be looking for robust methods to undertake state-of-the-art reviews—get started with SRs.

This chapter should also be of interest to more experienced empirical researchers who are not yet confident of the value of the SR methodology. Many European researchers are publishing SRs in software engineering, but relatively few researchers from the U.S. do so ([Kitchenham et al. 2009b], [Kitchenham et al. 2010a]).

I also hope that this chapter will alert empirical researchers to the possibility that their studies will contribute to subsequent SRs and that they consequently will report their results with future aggregation in mind. A recent SR found it impossible to undertake a full meta-analysis because the individual primary studies used very disparate practices for reporting their results ([Turner et al. 2008], [Turner et al. 2010]).

The aim of systematic reviews in the context of evidence-based software engineering is not just to provide a methodology for researchers; the aim is to influence practice. I hope, therefore, that managers and decision makers in industry also will find something in this chapter relevant to their needs. The main lesson for industry is that “common knowledge” and expert opinion should not be the sole basis for the decisions about the choice of software engineering methods. Furthermore, unfortunately, individual empirical studies cannot be trusted automatically. For important decisions concerning the adoption of new methods, decision makers need unbiased summaries of all relevant evidence, and SRs provide a means to deliver such summaries.