Factor analysis is another technique we can use to reduce dimensionality. However, factor analysis makes assumptions and PCA does not. The basic assumption is that there are implicit features responsible for the features of the dataset.
This recipe will boil down to the explicit features from our samples in an attempt to understand the independent variables as much as the dependent variables.
To compare PCA and factor analysis, let's use the iris dataset again, but we'll first need to load the factor analysis class:
>>> from sklearn.decomposition import FactorAnalysis
From a programming perspective, factor analysis isn't much different from PCA:
>>> fa = FactorAnalysis(n_components=2) >>> iris_two_dim = fa.fit_transform(iris.data) >>> iris_two_dim[:5] array([[-1.33125848, 0.55846779], [-1.33914102, -0.00509715], [-1.40258715, -0.307983 ], [-1.29839497, -0.71854288], [-1.33587575, 0.36533259]])
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you
Compare the following plot to the plot in the last section:
Since factor analysis is a probabilistic transform, we can examine different aspects such as the log likelihood of the observations under the model, and better still, compare the log likelihoods across models.
Factor analysis is not without flaws. The reason is that you're not fitting a model to predict an outcome, you're fitting a model as a preparation step. This isn't a bad thing per se, but errors here compound when training the actual model.
Factor analysis is similar to PCA, which was covered previously. However, there is an important distinction to be made. PCA is a linear transformation of the data to a different space where the first component "explains" the variance of the data, and each subsequent component is orthogonal to the first component.
For example, you can think of PCA as taking a dataset of N dimensions and going down to some space of M dimensions, where M < N.
Factor analysis, on the other hand, works under the assumption that there are only M important features and a linear combination of these features (plus noise) creates the dataset in N dimensions. To put it another way, you don't do regression on an outcome variable, you do regression on the features to determine the latent factors of the dataset.