The purpose of this section is to create a Scala class to be used in future chapters for validating models. For starters, the validation process relies on a set of metrics to quantify the fitness of a model generated through training.

trait Validation {
  def f1: Double
  def precisionRecall: (Double, Double)
}

object Label extends Enumeration {
  type Label = Value
  val TP, TN, FP, FN = Value
}

It is quite common that the labeled dataset used for both training and validation is not large enough. The solution is to break the original labeled dataset into K data groups. The data scientist creates K training-validation datasets by selecting one of the groups as a validation set then combining all other remaining groups into a training set as illustrated in the next diagram. The process is known as the K-fold cross validation [2:7].

The third segment is used as validation data and all other dataset segments except S3 are combined into a single training set. This process is applied to each segment of the original labeled dataset.

There is an obvious challenge in creating a model that fits both the training set and subsequent observations to be classified during the validation phase.

If the model tightly fits the observations selected for training, there is a high probability that new observations may not be correctly classified. This is usually the case when the model is complex. This model is characterized as having a low bias with a high variance. Such a scenario can be attributed to the fact that the scientist is overly confident that the observations he or she selected for training are representative to the real world.

The probability of a new observation being classified as belonging to a positive class increases as the selected model fits loosely the training set. In this case, the model is characterized as having a high bias with a low variance.

The mathematical definition for the bias, variance, and mean squared error (MSE) of the distribution are defined by the following formulas:

Note

Variance and bias for a true model, θ:

Mean square error:

Let's illustrate the concept of bias, variance, and mean square error with an example. At this stage, most of the machines learning techniques have not been introduced yet. Therefore, the example will emulate a multiple models fEst: Double => Double generated from non-overlapping training sets.

These models are evaluated against a test/validation datasets that are emulated by a model, emul. The BiasVarianceEmulator emulator class takes the emulator function and the size of the nValues validation test as parameters. It merely implements the formula to compute the bias and variance for each of the fEst models:

class BiasVarianceEmulator[T <% Double](emul: Double => Double, nValues: Int) {
    
  def fit(fEst: List[Double => Double]): Option[XYTSeries] = {
     val rf = Range(0, fEst.size)
     val meanFEst = Array.tabulate(nValues)( x =>  
         rf.foldLeft(0.0)((s, n) => s+fEst(n)(x))/fEst.size) // 1

     val r = Range(0, nValues)
     Some(fEst.map(fe => {
        r.foldLeft(0.0, 0.0)((s, x) => { 
          val diff = (fe(x) - meanFEst(x))/ fEst.size   // 2
          (s._1 + diff*diff, s._2 + Math.abs(fe(x)-emul(x)))} )
     }).toArray)
  }
}

The fit method computes the variance and bias for each of the fEst models generated from training. First, the mean of all the models are computed (line 1), and then used in the computation of the variance and bias. The method returns a tuple (variance, bias) for each of the fEst model.

Let's apply the emulator to three nonlinear regression models evaluated against validation data:

The client code for the emulator consists of defining the emul emulator function, and a list, fEst, of three models defined as tuples of (function, descriptor) of type (Double=>Double, String). The fit method is call on the model functions extracted through a map, as shown in the following code:

val emul = (x: Double) => 0.2*x*(1.0 + Math.sin(x*0.05))
val fEst = List[(Double=>Double, String)] (
  ((x: Double) => 0.2*x, "y=x/5"),
  ((x: Double) => 0.0003*x*x + 0.18*x, "y=3e-4.x^2-0.18x"),
  ((x: Double) =>0.2*x*(1+Math.sin(x*0.05),
                "y=x(1+sin(x/20))/5"))
val emulator = new BiasVarianceEmulator[Double](emul, 200)
emulator.fit(fEst.map( _._1)) match {
  case Some(varBias) => show(varBias)
  case None => …
}

The JFreeChart library is used to display the test dataset and the three model functions.

Fitting models to dataset

The variance-bias trade-off is illustrated in the following scatter chart using the absolute value of the bias:

The more complex the function, the lower the bias is. It is usually, but not always related to, a high variance. The most complex function y=x (1+sin(x/20))/5 has by far the highest variance and the lowest bias. The more complex model matches fairly well with the training dataset. As expected, the mean square error reflects the ability of each of the three models to fit the test data.

Mean square error bar chart

The low bias of the complex model reflects in its ability to predict new observations correctly. Its MSE is therefore low, as expected.

Complex models with low bias and high variance are known as overfitting. Models with high bias and low variance are characterized as underfitting.

The methodology presented in the example can be applied to any classification and regression model. The list of models with low variance includes constant function and models independent of the training set. High degree polynomial, complex functions, and deep neural networks have high variance. Linear regression applied to linear data has a low bias, while linear regression applied to nonlinear data has a higher bias [2:8]

Overfitting affects all aspects of the modeling process negatively, for example:

However, there are well-proven solutions to reduce overfitting [2:9]: