In a way similar to what we have seen for clustering evaluation, when dealing with classification models, we look for closeness to perfection. Every element of the population, and more frequently of a separate validation dataset, is provided with the correct theoretical classification, tagged, and, subsequently, the actual classification is compared with the one provided from the model. The more the model output coincides with the actual classification, the more accurate it is. A useful instrument when dealing with classification evaluation is the so-called confusion matrix, which is a double-entry table useful for understanding the level of performance of the model for a given category of classification.
The confusion matrix shows, on one entry, a true and a false for the model, and on the other entry a true and a false for the actual classification. We therefore have four possible combinations:
- True positive: Number of cases for which both the model and the theoretical classification show the given attribute. These are the cases where the model correctly performs.
- True negative: Number of cases where the attribute is not shown from both the model and the theoretical classification. Here again, the model is correctly performing.
- False positive: The model here shows the presence of the attribute while the theoretical classification doesn't. We count here a first type of error for the models.
- False negative: The opposite of the previous one, since the model here doesn't mark the presence of the attribute while the theoretical classification does.
Starting from the number shown within the confusion matrix, we can easily compute a comprehensive measure of the model performance: the accuracy.
The accuracy is measured as follows:
accuracy = (Σ True positive + Σ True negative)/Σ Total population
As can be easily understood, we see here a measure of how many times the model assigns the right classification within a given population.