The quote at the beginning of this chapter mentions using ensembles to win machine learning competitions. However, they do have practical applications. I've provided a definition of what ensemble modeling is, but why does it work? To demonstrate this, I've co-opted an example from the following blog, which goes into depth at a number of ensemble methods: http://mlwave.com/kaggle-ensembling-guide/.
As I write this chapter, we're only a day away from the 2018 College Football Championship—the Clemson Tigers versus the Alabama Crimson Tide. Let's say we want to review our probability of winning a friendly wager where we want to take the Tide minus the points (5.5 points at the time of writing).
Assume that we've been following three expert prognosticators who said, All have the same probability of predicting that the Patriots will cover the spread (60%). Now, if we favor any one of the so-called experts, it's clear that we have a 60% chance of winning. However, let's see what creating an ensemble of their predictions can do to increase our chances of profiting and humiliating friends and family.
Start by calculating the probability of each possible outcome for the experts picking Alabama, and let's assume that the probability is the same at 60%. If all three pick Alabama, we have 0.6 x 0.6 x 0.6 or a 21.6% chance that all three are correct.
If any two of the three pick Alabama, then we have (0.6 x 0.6 x 0.3) x 3 for a total of 43.2%.
By using majority voting, if at least two of the three pick Alabama, then our probability of winning becomes almost 65% (21.6 + 43.2), which is an absolute improvement of 5%.
This is a rather simplistic example but representative nonetheless. In machine learning, it can manifest itself by incorporating the predictions from several OK or even weak learners to improve overall accuracy. The diagram that follows shows how this can be accomplished:
In this graphic, we build three different classifiers and use their predicted probabilities as input values to a fourth and different classifier in order to make predictions on the test data. Let's see how to apply this with R.