In the previous section I discussed some SRs that offer new insights into software engineering. Many other SRs and mapping studies over the past few years have covered a variety of topics (see [Kitchenham et al. 2009b], [Kitchenham et al. 2010a], the IST virtual special issue,[10] and Chapter 12). I believe these studies should start to change the way we do research in software engineering. Anyone interested in a particular topic needs to check for SRs addressing the topic. If no SRs exist, it may be a good idea to do one. If there are existing reviews, they can act as the starting point for your own literature search or can point you to topic areas where new research is necessary.
However, SRs have obvious limitations. First, just because something claims to be an SR doesn’t mean it was necessarily a high-quality review. You should read SRs as critically as you read any other research paper. You can use Greenhalgh’s evaluation criteria [Greenhalgh 2000] or the five criteria used by the Centre for Reviews and Dissemination [Centre for Reviews and Dissemination 2007]:
Are the review’s inclusion and exclusion criteria described and appropriate?
Is the literature search likely to have covered all relevant studies?
Were the studies synthesized?
Did the reviewers assess the quality/validity of the included studies?
Were the basic data/studies adequately described?
The second major problem is that SRs rely on the availability of high-quality primary studies. The studies discussed in the previous section suggest that there are a relatively large number of primary studies but cast doubts on their quality. For SRs, we need primary studies that:
Conform to best quality guidelines for the type of study being undertaken
Report their results in sufficient detail for meta-analysis to be performed
Are independent of one another in terms of research groups and research materials (in contrast to Basili’s et al.’s suggestion for families of experiments [Basili et al. 1999])
Collect data concerning possible moderating variables, e.g., subject type and experience, task complexity, size, and duration
Furthermore, even if our primary studies of human-centric methods adopt these best practices, I remain to be convinced that meta-analysis-based aggregation can reliably assess the impact of a method/technique unless we are able to use professional subjects in realistic situations (i.e., doing tasks of realistic complexity and duration).
Nonetheless, even if there are problems with primary studies and the interpretation of the results of meta-analyses, it seems to me to be pointless to undertake empirical studies if we don’t attempt to organize results into an empirical body of knowledge about our methods and techniques. Furthermore, we need to adopt the discipline of SRs to ensure that we aggregate our results as fairly and openly as possible.