Although alpha is generally the preferred method of estimating reliability (particularly in the social sciences), it is a fairly basic method that can be easily influenced. It is not robust with respect to many potential characteristics of a sample or instrument. It is important to be aware of these characteristics so that appropriate steps can be taken to reduce their effect and improve our estimates of reliability. Each characteristic is described below.

High average inter-item correlation. All other things being equal, alpha is higher when the average correlation between items is higher. But even Cronbach specifically pointed out that when inter-item correlations are low, alpha can be high with enough items with low intercorrelations. This is one of the chief drawbacks to interpretability of alpha—that with enough mostly unrelated items, alpha will move into the “reasonable” range that most researchers use as a rule of thumb.

Length of the scale. As mentioned above, all other things being equal, longer scales will have higher alphas.

Reverse coded items (negative item-total correlations). Many scales are constructed with reverse-coded items. However, alpha cannot provide accurate estimates when the analysis includes items with negative item-total correlations. Thus, any item that is expected to have a negative item-total correlation (e.g., if the factor loading is negative when most others are positive) should be reversed prior to analysis.

Random responding or response sets. Random responding (discussed in Chapter 8) tends to attenuate all of these estimates because it primarily adds random error. Thus, failure to identify this issue in your data will lead to under-estimation of the internal consistency of the data. Response sets can have a variety of effects, depending on the response set. Some types of response sets will inflate alpha estimates and some can attenuate alpha (for an overview of response sets, and how one can identify them, you might see Osborne & Blanchard, 2011).

Multidimensionality. The assumption of alpha is that all items within a particular analysis represent a single dimension, or factor. To the extent that assumption is violated, the estimate of alpha will be misestimated. Thus, the factor structure of the scale should be considered before submitting items to this type of analysis.

Outliers. Outliers (inappropriate values, also discussed in Chapter 8 usually have the effect of increasing error variance. This generally has the effect of attenuating the estimate of alpha. Thus, data should be carefully screened prior to computing alpha.

In addition to the above factors that can influence the estimate of alpha, a theoretical issue can hinder its interpretation. Alpha was built for a time when researchers often summed or averaged items on scales. Many researchers do this today. Of course, when summing or averaging items in a scale, you are making an assumption that all items contribute equally to the scale—that the weighting of each item is identical. Alpha also assumes that all items contribute equally. Yet from what we have seen in earlier chapters, that might not be a valid assumption, and it is not the assumption made by EFA (or confirmatory factor analysis, latent variable modeling, IRT, Rasch measurement, etc.). For example, in Chapter 4 we saw that the pattern loadings for the engineering problem-solving items ranged from 0.79 to 0.91, and for the GDS the loadings ranged from 0.23 to 0.68 when only one factor was extracted. If you square the loadings to estimate the unique proportion of the shared variance that is attributable to each item, this amounts to a range of 0.62 to 0.82 (for engineering problem-solving) and from 0.05 to 0.46 for GDS. Thus, each item contributes a different proportion of variance to each factor.

Historically, this led researchers to create “factor scores,” which weighted each item by the factor loading to more closely approximate what a latent variable score might be. As we discussed in Chapter 9, we do not believe this is a good idea. However, there is not currently a good way to deal with this theoretical issue when using alpha (in our opinion). Those interested in estimating internal consistency via alpha must simply be aware of its violation of the assumption of unique item contribution and live with it.