Ensemble learning techniques are basically solutions developed to join together predictions coming from different models. The three most basic techniques to accomplish this task are:

• Averaging prediction: Which applies to regression problems and basically involves computing a final prediction equal to the average of predictions coming from single models.
• Majority vote: Which is applicable to classification problems involving the definition of a final prediction equal to the prediction expressed from the higher number of models, that is, the mode within the available predictions.
• Weighted average or weighted majority vote: Which are applicable to both regression and classification problems, involving the computation of a weighted average for the case of regression problems or the determination of a different computed mode assigning different relevance to votes expressed from different models. The weight assigned to each model is usually proportional to the accuracy shown from the models themselves.

You should be aware that other, more advanced techniques are available that sometimes are needed to obtain better results. For the sake of completeness, I will list the most popular of them here:

• Bagging, which is basically equal to a majority vote, except for the fact that different models are fitted on random subsets of the whole estimation population
• Boosting, which involves iteratively building an ensemble model, each time starting from the misclassification produced from the previous model
• Stacking, which involves training a new model starting from the outputs produced from the other models

I will provide you with some good references as usual, in case you want to deepen your knowledge of the topic. Let's now apply basic ensemble learning techniques to our data.