First, let's start with the definition of the layer class, MLPLayer, which is completely specified by its position in the network and the number of nodes it contains:

class MLPLayer(val id: Int, val len: Int) {
   val output = new DblVector(len) //1
   val delta = new DblVector(len) //2
...output.update(0, 1.0) //3

The id parameter is the order of the layer (0 for input, 1 for the first hidden layer,…, n-1 for the output layer) in the network. The len value is the number of elements or nodes, including the bias element, in this layer. The output vector for the layer (line 1) is an uninitialized vector of values updated during the forward propagation, except for the first value (bias element), which is set to 1 (line 3). The delta vector associated to the output vector (line 2) is updated through the error backpropagation algorithm, described in the next section.

The output values, except the bias element, is initialized using the set method:

def set(x: DblVector): Unit = x.copyToArray(output,1)

A synapse is defined as a pair of real values:

Its type is defined as MLPSynapse, as shown here:

type MLPSynapse = (Double, Double)

A connection between two consecutive layers implements the matrix of synapses, the (w_ij, w_ij) pairs. The MLPConnection instance is created with the following parameters:

The MLPConnection class is defined as follows:

class MLPConnection(config: MLPConfig, src: MLPLayer, dst: MLPLayer)

The last step in the initialization of the MLP algorithm is the selection of the initial (usually random) values of the weights (synapse). The following code snippet initializes the weights for non-bias neurons as random values in the range [0, beta] with beta <= 1.0.

The weight for the bias is obviously defined as w₀=+1, and its weight adjustment is initialized as w₀ = 0, as shown here:

Val beta = 0.1
val synapses = Array.tabulate(dst.len)(n => 
   if(n > 0) Array.fill(src.len)((beta*Random.nextDouble, 0.0))
   else Array.fill(src.len)((1.0, 0.0))
)

Once the topology, synapses, layers, and connections of the MLP algorithm are defined, the initialization of the MLPModel model is straightforward:

val layers = topology.zipWithIndex
                     .map(t => MLPLayer(t._2, t._1+1))
val connections = Range(0, layers.size-1).map(n =>
    new MLPConnection(config, layers(n), layers(n+1))).toArray

The layers are created by traversing the network topology and instantiating each layer with its proper index and number of elements. The connections are instantiated by selecting two consecutive layers of index n (with respect to n+1) as source (with respect to destination).

Once initialized, the MLP model is ready to be trained using a combination of forward propagation, output error back propagation, and iterative adjustment of weights and gradients of weights.

As mentioned earlier, the output values of a hidden layer are computed as a logistic function (the activation function) of the dot product of the weights w_ij and the input values x_i.

In the following diagram, the MLP algorithm computes the linear product of the weights w_ij and input x_i for the hidden layer. The product is then processed by the activation function (sigmoid or hyperbolic tangent). The output values z_j are then combined with the weights v_ij of the output layer. The output layer doesn't have an activation function.

The mathematical formulation of the output of a neuron j is defined as a composition of the activation function and the dot product of the weights w_ij and input values x_i.

Note

Computation of the output y for the output layer:

Estimation of the output values for binary classification with an activation function :

As seen in the network architecture section, the output values for the multinomial (or multiclass) classification with more than two classes are normalized using an exponential function (softmax).

The computational model

The computation of the output values y from the input x is known as the input forward propagation. For the sake of simplicity, we represent the forward propagation between layers with the following block diagram. Such a representation will be quite convenient for the design and implementation of the MLP.

A computation model of input forward propagation

This diagram illustrates a computational model for the input forward propagation, as the programmatic relation between the source and destination layers and their connectivity. The input x is propagated forward through each connection.

The connectionForwardPropagation method computes the dot product of the weights and the input values, and applies the activation function in the case of hidden layers, for each connection. Therefore, it is a member of the MLPConnection class.

The forward propagation of input values across the entire network is managed by the MLP algorithm itself.

The forward propagation of the input value is used in the classification or prediction y =f(x). It depends on the value weights w_ij and v_ij that need to be estimated through training. As you may have guessed, the weights define the model of a neural network similar to the regression models. Let's look at the connectionForwardPropagation method of the MLPConnection class:

def connectionForwardPropagation: Unit = {
  val synps= synapses.drop(1)
  val _output = synps.map(x => { //1
      val sum = x.zip(src.output)
                  .foldLeft(0.0)((s, xy) => s + xy._1._1*xy._2)
      if(!isOutLayer) config.activation(sum) //2
      else sum
  })
  val out = if(isOutLayer) mlpObjective(_output) else _output //3
  out.copyToArray(dst.output, 1)     
}

The first step is to compute the linear dot product of the _output output of the current source layer, src, for this connection, and the weights, w (line 1). The activation method, the implementation of which is described in the next paragraph, is applied to the dot product, dot (line 2). If the destination layer of the connection is the output layer, then the output values are processed according to the mlpObjective objective of the algorithm (line 3).

trait MLPObjective { def apply(y: DblVector): DblVector }

class MLPBinClassifier extends MLPObjective {
  override def apply(y: DblVector): DblVector = output
}

class MLPMultiClassifier extends MLPObjective {
   override def apply(y:DblVector):DblVector = softmax(y.drop(1))
   def softmax(y: DblVector): DblVector = { …}
}

Softmax

In case of a classification problem with K classes (K > 2), the output has to be converted into a probability [0, 1]. For problems that require a large number of classes, there is a need to boost the output y_k with the highest value (or probability). This process is known as the exponential normalization or softmax [9:8].

Note

Softmax formula for multinomial (K > 2) classification is as follows:

Here is the simple implementation of the softmax method of the MLPMultiClassifier class:

def softmax(y: DblVector): DblVector = {
   val softmaxValues = new DblVector(y.size)
   val expY = y.map( Math.exp(_))//1
   val expYSum = expY.sum
   expY.map( _ /expYSum).copyToArray(softmaxValues, 1) //2
   softmaxValues
}

First, the output values are transformed to exponential, expY (line 1). The exponentially transformed outputs are then normalized by their sum, expYSum, to generate the array of softmaxValues output (line 2). Once again, you do not have to update the bias element y(0).

The second step in the training phase is the back propagation of the output error.

Once the input features are propagated across the neural network, the sum of squared errors, sse, for the output layer of the MPLayer type is computed at each epoch, as follows:

def sse(labels: DblVector): Double = {
   var _sse = 0.0
   output.drop(1)  //1
        .zipWithIndex
        .foreach(on => {
     val err = labels(on._2) - on._1  //2
     delta.update(on._2+1, on._1* (1.0- on._1)*err) //3
     _sse += err*err
  })
  _sse*0.5  //4
}

As expected, the computation of the sum of squared errors requires the labeled values, labels, and the objective method as arguments. The vector output values, output, stripped of the bias node (line 1) is used to compute the difference, err, between the label and the actual output (line 2). The delta value (line 3), described in the next section, is used in the back-propagation algorithm to adjust the weights of the output and hidden layers. Note that the sum of squares, _sse, is divided by 2 (line 4), so its derivative is err.

The error backpropagation is an algorithm that estimates the error for the hidden layer in order to compute the change in weights of the network. It takes the sum of squared errors on the output as input.

Error propagation

The objective of the training of a perceptron is to minimize the sum of squared errors at the output layer. The error for each output neuron, y_k, is computed as the difference between a predicted output value and label output value. This approach does not work for the hidden layers z_j because the label value is unknown.

The partial derivative of the sum of squared output error over each weight of the output layer is computed as the composition of the derivative of the square function, and the derivative of the dot product of weights and the input z.

Note

Derivative of the output SSE over the weighs of the output layer:

As mentioned earlier, the computation of the partial derivative of the sum of squared error over the weights of the hidden layer is a bit tricky. Fortunately, the partial derivative can be broken down into the following three pieces using the output layer values and the output of the hidden layer:

• Derivative of sum of squared error over the output value y_k
• Derivative of the output value y_k over the hidden value z_j knowing that the derivative of a sigmoid is (1 - )
• Derivative of the output of the hidden layer z_j over the weights w_ij

Note

Derivative of error over the weights of the hidden layer:

The computational model

The computational model for the error backpropagation algorithm is very similar to the forward propagation of the input. The main difference is that the propagation of the derivative delta is performed from the output layer to the input layer. The following diagram illustrates the computational model of the backpropagation in the case of two hidden layers z_s and z_t:

The connectionBackPropagation method propagates the error back to the previous layer. It is a member of the MLPConnection class. The backpropagation of the output error across the entire network is managed by the MLP class.

It implements the two set of equations where synapses (j)(i)._1 are the weights w_ji, dst.delta is the vector of error derivative in the destination or next layer, and src.delta is the error derivative on the outputs in the source (or antecedent) layer, as shown here:

def connectionBackpropagation: Unit =  
  Range(1, src.len).foreach(i => {
    val dot = Range(1, dst.len).foldLeft(0.0)((s, j) => 
                         s + synapses(j)(i)._1*dst.delta(j)) //1
    src.delta(i) = src.output(i)*(1.0 - src.output(i))*dot//2
})

The dot product of the synapse weights and the errors of the destination layers (line 1) is used to compute the delta on the source (or previous layer) layers (line 2).

The connection weights v and w are adjusted by computing the sum of the derivative of the error, over the weights scaled with a learning factor. The gradient of weights are then used to compute the error of the output of the source layer [9:9].

Momentum factor for gradient descent

The simplest algorithm to update the weights is the gradient descent [9:10].

The gradient descent is a very simple and robust algorithm. However, it is slower in converging toward a global minimum than the conjugate gradient or the quasi-Newton method (refer to the Summary of optimization techniques section in Appendix A, Basic Concepts).

There are several methods available to speed up the convergence of the gradient descent toward a minimum: momentum factor and adaptive learning coefficient [9:11].

Large variations of the weights (or large value of the gradient of weights) cause the gradient descent to require more training iteration in order to converge. This is particularly true for a training strategy known as online training. The training strategies are discussed in the next section. The momentum factor is used for the remaining section of the chapter.

Note

The computation of neural network weights using gradient descent is as follows:

The computation of neural network weights using gradient descent method with momentum coefficient is as follows:

The basic gradient descent algorithm is selected by setting the momentum factor to zero.

def connectionUpdate: Unit =  
  Range(1, dst.len).foreach(i => {  
    val delta = dst.delta(i) //1

    Range(0, src.len).foreach(j => { 
       val _output = src.output(j) //2
       val oldSynapse = synapses(i)(j)
       val grad = config.eta*delta*_output //3
       val deltaWeight = grad + config.alpha* oldSynapse._2 //4
       synapses(i)(j) = (oldSynapse._1 + deltaWeight, grad) //5
    })
  })

The convergence criterion consists of evaluating the sum of squared errors against a predetermined threshold eps. It is common to normalize the sum of squared errors by the number of observations.

The MLPConfig configuration of the multilayer perceptron consists of the definition of the network configuration with hidden layers, the learning parameters, the training parameters, and the activation function:

Class MLPConfig(val alpha: Double, val eta: Double, val hidLayers: Array[Int], val numEpochs: Int,val eps: Double,val activation: Double=>Double) extends Config

For the sake of readability, the name of the configuration parameters matches the symbols defined in the mathematical formulation:

The five steps of the training cycle have been implemented for each connection or matrix of synapses (weights, gradient of weights). The management of the cycle is performed by the algorithm defined by the MLP class, as shown here:

class MLP[T <% Double](config: MLPConfig, xt: XTSeries[Array[T]], labels: DblMatrix)(implicit val mlpObjective: MLP.MLPObjective) 
extends PipeOperator[Array[T], DblVector] {
   val model: Option[MLPModel]
   def |> : PartialFunction[Array[T], DblVector]
}

The MLP algorithm takes the following parameters:

The five steps of the training cycle or epoch is summarized in the following diagram:

Let's apply the five steps of a training epoch in a trainEpoch method of the MLPModel class using a simple the foreach Scala iterator, as shown here:

def trainEpoch(x: DblVector, y: DblVector): Double = {
   inLayer.set(x)

   connections.foreach( _.connectionForwardPropagation) //1
   val _sse = sse(y) //2
   val bckIterator = connections.reverseIterator 
   bckIterator.foreach( _.connectionBackpropagation) //3
   connections.foreach( _.connectionUpdate) //4
  _sse
}

You can certainly recognize the first four stages of the training cycle: forward propagation of the input, x (line 1), computation of the sum of squared errors, _sse (line 2), the back propagation of the error (line 3), and the recomputation of the weight and gradient of weight associated with each synapse (line 4).

One important remaining issue is finding a strategy to conduct the training of time series, as ordered sequences of data. There are two strategies to create an MLP model for time series:

An online training is faster than batch training because the convergence criterion has to be met for each data point, possibly resulting in a smaller number of epochs [9:12]. Techniques such as the momentum factor, which is described earlier, or any adaptive learning scheme improve the performance of the online training process further.

The online training strategy is applied to a financial time series for the remainder of this chapter.

There are two approaches to find the most appropriate network architecture for a given classification or regression problem; they are:

The destructive tuning strategy removes the synapses by zeroing out their weights. This is commonly accomplished by using regularization.

You have seen that regularization is a powerful technique to address overfitting in the case of the linear and logistic regression in the The ridge regression section in Chapter 6, Regression and Regularization. Neural networks can benefit from adding a regularization term to the sum of squared errors. The larger the regularization factor is, the more likely some weights will be reduced to zero, thus reducing the scale of the network [9:13].

The model instance is created (trained) during the instantiation of the multilayer perceptron. The model is created by iterating the training cycle over all the data points of the time series xt, and through multiple epochs until the total sum of squared errors is smaller than the threshold eps, as in the following code:

var converged = false
val model: Option[MLPModel] = {
   val _model = new MLPModel(config, xt(0).size, labels(0).size)(mlpObjective)  //1
   val errScale = 1.0/(labels(0).size*xt.size)  //4

   converged = Range(0, config.numEpochs).find( _ => {
     xt.toArray.zip(labels)
               .foldLeft(0.0)((s, xtlbl) => 
                   s + _model.trainEpoch(xtlbl._1, xtlbl._2) //2
                )*errScale < config.eps  //3
   }) != None
   _model
}

The model is first initialized (line 1). The first four stages of the MLP training cycle are executed by the MLPModel.trainEpoch method described in the previous section (line 2). The method returns the sum of squared errors for each observation in the time series. The sum of squared errors for the observations are summed, then evaluated against the convergence criterion, eps (line 3). The sum of squared errors is normalized for the size of the time series and the size of the output vector (line 5). The implementation uses the Scala method, find, to exit from the iterative loop before the maximum number of epochs, config.numEpochs, is reached.

Once the model is created during the instantiation of the multilayer perceptron, it is available to predict the class of a new observation.

The prediction method of the MLPModel class, getOutput, takes a new observation (feature vector) as argument and returns the output by using the forward propagation algorithm:

def getOutput(x: DblVector): DblVector = {
  inLayer.set(x)
  connections.foreach( _.connectionForwardPropagation)
  outLayer.output
}

The classification method is implemented as the data transformation |>. It returns the predicted value, normalized as a probability if the model was successfully trained; None, otherwise:

def |> : PartialFunction[Array[T], DblVector] = {
  case x: Array[T] if(model!=None && x.size == dimension(xt)) =>
    model.get.getOutput(x))
}

Our MLP class is now ready to tackle some classification challenges.