While intriguing, and perhaps conceptually sensible, these results are not definitive, and the replicability of second-order factors is not well-studied. It is also possible (or likely) that this scale should have been a single-factor scale from the beginning. As mentioned previously, the best way to test this competing hypothesis is to perform confirmatory factor analysis comparing model fit between the three-factor model, the three-factor model with a higher-order factor, and a one-factor model, using a different, large sample. Then, and only then, would we be able to draw conclusions about the best fitting model for this data. For now, let us take a simple example of replication and see how well these results might replicate. Unfortunately, this original sample is relatively small, so we randomly selected three samples of N=150 each (with replacement) from the original sample, and performed the same analysis as above to explore whether, in this particular case, replication of second-order factors is a reasonable expectation. There are endless caveats to what we are about to do, including the lousy sample size. Let’s explore anyway!

As you can see in Communalities from second-order factor analysis, the initial communalities were neither terribly close nor terribly far off. Remember that these represent percent variance accounted for—for example, factor 2 ranges from about 20% accounted for to about 31%, a wide margin. Likewise, the extracted communalities varied a bit. The extracted eigenvalues were 1.42, 1.29, and 1.24 (representing 47.3%, 43.0%, and 41.3% variance accounted for, respectively). Factor loadings from second-order factor analysis shows the factor matrix from the three analyses. As you can see, they are not terribly different, but not terribly similar.

If we were to conduct this analysis in practice, we would use the factor loadings table to construct the meaning of the higher-order factor. Reviewing the results in the table above, this is a little concerning. Across the three samples, the order of importance of the three factors changes. In the first and third sample, the third factor is the most important and the second factor is the least. In the second sample, the order almost reversed, with the second factor as the most important and the first factor as the least. This might be nitpicky, because we don’t think EFA applied for this purpose is a best practice. If you are going to use this, we hope you replicate or bootstrap to give the reader an idea of how robust or precise your estimates are.