After you are done with the exploratory factor analysis, your journey is just beginning, rather than ending. The exploration phase might be drawing to a close, but the psychometric evaluation of an instrument is merely starting. The process of performing exploratory factor analysis is usually seeking to answer the question of whether a given set of items forms a coherent factor (or several factors). After we decide whether this is likely, evaluating how well those constructs are measured is important. Along the way, we can also ask whether the factor that is being examined needs all the items in order to be measured effectively.

To fully evaluate an instrument, we should evaluate whether the factors or scales that we derive from the EFA are reliable, confirmed in a new sample, and stable (invariant) across multiple groups. In this chapter, we will briefly look at the most common method of assessing scale reliability, Cronbach’s coefficient alpha.

Let us start with a discussion of the modern view of reliability and validity. When developing a scale to be used in research, there is a delicate dance between focusing on creating a scale that is a “good” scale, and the acknowledgment in modern research methods that things like factor structure, reliability, and validity are joint properties of a scale and of the particular sample being used (Fan & Thompson, 2001; Wilkinson, 1999). It should be self-evident to modern researchers that a scale needs to be well-developed in order to be useful, and that we do that in the context of a particular sample (or series of samples, as we recommended when discussing replication). Thus, those of us interested in measurement must hold two somewhat bifurcated ideas in mind simultaneously —that a scale can be stronger or weaker, and that scales are strong or weak only in the context of the particular sample being used. This can lead to a nihilistic mindset if carried too far, and so we recommend that we take a moderate position in this discussion: that scales can be more or less strong, but that all scales need to be evaluated in the particular populations or data that they reside in.