The conditional probability in a linear regression model is a linear function of its weights [6:13]. The logistic regression model addresses the nonlinear regression problem by defining the logarithm of the conditional probability as a linear function of its parameters.

First, let's introduce the logistic function and its derivative, which are defined as follows:

Have a look at the following graph:

The graph of the logistic function and its derivative

The remainder of this section is dedicated to the application of the multivariate logistic regression to a binary classification (two classes).

The logistic regression is popular for several reasons; some are as follows:

Let's consider the classification problem using two classes. As discussed in the Validation section of Chapter 2, Hello World!, even the best classifier produces false positives and false negatives. The training procedure for a binomial classification is illustrated in the following diagram:

Illustration of the binomial classification for a 2-dimension dataset

The purpose of the training is to compute the hyperplane that separates the observations into two categories or classes. Mathematically speaking, a hyperplane in an n-dimensional space (number of features) is a subspace of n-1 dimensions. The separating hyperplane of a three-dimension space is a curved surface. The separating hyperplane of a two-dimension problem (plane) is a line. In our preceding example, the hyperplane segregates/separates a training set into two very distinct classes (or groups), class 1 and class 2, in an attempt to reduce the overlap (false positive and false negative).

The equation of the hyperplane is defined as the logistic function of the dot product of the regression parameters (or weights) and features.

The logistic function accentuates the difference between the two groups of training observations, separated by the hyperplane. It pushes the observations away from the separating hyperplane towards either of the classes.

In the case of two classes, c1 and c2 with their respective probabilities, p(C=c1| X=xⁱ|w) = p(xⁱ|w) and p(C=c2 |X= xⁱ|w) = 1- p(xⁱ|w), where w is the model parameters set or weights in the case of the logistic regression, the following functions can be defined:

Note

The log likelihood:

Conditional probabilities using the logit function:

The log likelihood for the binomial logistic regression:

First order derivative for the log likelihood:

Let's implement the logistic regression without a penalty term using the Apache Commons Math library. The library contains several least squares optimizers, allowing you to specify the minimizing algorithm, optimizer, for the loss function in the logistic regression class LogisticRegression:

class LogisticRegression[T <% Double](xt: XTSeries[Array[T]], labels: Array[Int], optimizer: LogisticRegressionOptimizer) extends PipeOperator[Array[T], Int]{
  val model: Option[RegressionModel] = { … }
  …
}

The parameters of the logistic regression class are the multivariate time series (features) xt, the target or labeled data, labels, and the optimizer algorithm used to minimize the loss function or residual sum of squares. In the case of the binomial logistic regression, labels are assigned the values of 1 for one class and 0 for the other.

The purpose of the training is to determine the regression coefficient, model._1, which minimizes the loss function. The residual sum of squares (RSS) is computed as model._2.

For convenience, the definition and the configuration of the optimizer are encapsulated in the LogisticRegressionOptimizer class.

The implementation of the logistic regression uses the following components:

The key software components of the logistic regression are described in the following UML class diagram:

The UML class diagram for the logistic regression

Our implementation of the training of the logistic regression model leverages either the Gauss-Newton or the Levenberg-Marquardt nonlinear least squares optimizers, (refer to the Nonlinear least squares minimization section in Appendix A, Basic Concepts) packaged with the Apache Commons Math library.

The training of the logistic regression is performed by the train method:

val model: Option[RegressionModel] = train

The implementation of the training method, train, relies on the following five steps:

The workflow and the Apache Commons Math classes used in the training of the logistic regression are visualized by the following flow diagram:

The workflow for training the logistic regression using Apache Commons Math

The first four steps are required by the Apache Commons Math library to initialize the configuration of the logistic regression prior to the minimization of the loss function. Let's start with the configuration of the least squares optimizer.

class LogisticRegressionOptimizer(maxIters: Int, maxEvals: Int,eps: Double, lsOptimizer: LeastSquaresOptimizer){
  def optimize(lsProblem: LeastSquaresProblem): Optimum = lsOptimizer.optimize(lsProblem)
}}

final val initWeight = 0.5
val weights0 = Array.fill(xt(0) +1)(initWeight) //1

val lrJacobian = new MultivariateJacobianFunction {
  override def value(w:RealVector):Pair[RealVector,RealMatrix] ={
    val _w = w.toArray
    val gradient = xt.toArray
                     .map( g => {  //2
         val expn = g.zip(_w.drop(1))
                   .foldLeft(_w(0))((s,z) => s + z._1*z._2)
       val logIt = 1.0/(1.0 + Math.exp(-expn)) //3
       (logIt, logIt *(1- logIt))  //4
    })
   
   val jacobian = Array.ofDim[Double](xt.size, weights0.size)//5
   xt.toArray.zipWithIndex.foreach(xi => {  //6
      val df: Double = gradient(xi._2)._2
      Range(0, xi._1.size).foreach(j => 
                            jacobian(xi._2)(j+1) = xi._1(j)*df)
      jacobian(xi._2)(0) = 1.0  //7
   })
   (new ArrayRealVector(gradient.map(_._1)), 
       new Array2DRowRealMatrix(jacobian))  //8
  }
}

val exitCheck = new ConvergenceChecker[PointVectorValuePair] {
   override def converged(iteration: Int, prev: PointVectorValuePair, current:PointVectorValuePair): Boolean =  {
     val delta = prev.getValue
                     .zip(current.getValue)
                     .foldLeft(0.0)((s, z) =>{ 
                         val d = z._1 - z._2
                         s + diff*diff
                      })
   Math.sqrt(delta)<optimizer.eps && iteration>=optimizer.maxIters
  }
}

val builder = new LeastSquaresBuilder
val diagWeights0 = Array.fill(xt.size)(1.0) //1
val wMatrix = MatrixUtils.createRealDiagonalMatrix(diagWeights0)
val lsp = builder.model(lrJacobian)  //2
                 .weight(wMatrix) 
                 .target(labels)  //7
                 .checkerPair(exitCheck) //5
                 .maxEvaluations(optimizer.maxEvals) //3
                 .start(weights0) //6
                 .maxIterations(optimizer.maxIters) //4
                 .build

val optimum = optimizer.optimize(lsp)
(optimum.getPoint.toArray, optimum.getRMS)

val deltaPrice = prices.drop(1).zip(prices).dropRight(1)) .map(p => if(p._1>p._2) 1 else 0)

val MAXITERS = 80; val MAXEVALS = 1000; val EPS = 1e-4

val lsOptimizer = LogisticRegressionOptimizer(MAXITERS, MAXEVALS, EPS, new LevenbergMarquardtOptimizer)
val xt = XTSeries[DblVector](features)
val regression = new LogisticRegression[Double](xt, deltaPrice, lsOptimizer)
val rms = regression.rms.get
val weights = regression.weights.get

As mention earlier and despite its name, the binomial logistic regression is a binary classifier. The classification method is implemented as a data transformation by overriding the pipe operator:

type Feature = Array[T]
final val MARGIN = 0.01
def |> : PartialFunction[Feature, Int] = { //1
  case x: Feature if(model!=None && model.get.size-1==x.size) =>{
    val w = _model.get.weights
    val dot = x.zip(w.drop(1))
               .foldLeft(w(0))((s,xw) => s + xw._1*xw._2)//2
   if(logit(dot) > 0.5 + MARGIN) 1 else 0 //3
  }
}

The classification method, |>, checks if the number of model parameters (weights) is equal to the number of features plus 1 (line 1) and throws an exception if the test fails. The dot product of the weights and the features is computed using a fold. Finally, the method returns 1 (class 1, which signifies that the price variation of the ETF is positive) if the value of the sigmoid is greater than 0.5. It returns 0 otherwise (class 2, which signifies that the price variation of the ETF is negative) (line 3).

Let's apply the classification to the original training set, features, to validate our model (weights):

val predicted = features.map(x => regression |> x)

The direction of the price variation of the Copper ETF, price(t+1) – price(t), is compared to the direction predicted by the logistic regression. The result is plotted with the success value if the positive or negative direction is correctly classified, otherwise, it is plotted with the failure value:

The logistic regression was able to classify 78 out of 121 trading sessions (65 percent accuracy).

Now, let's use the logistic regression to predict the positive price variation for the Copper ETF, given its volatility and trading volume. This trading or investment strategy is known as being long on the market. This use case ignores the trading sessions for which the price was either flat or declined:

The logistic regression was able to correctly predict the positive price variation for 58 out of 64 trading sessions (90.6 percent accuracy). What is the difference between the first and second test cases?

In the first case, the separating hyperplane equation, w0 + w1.volatility + w2.volume, is used to segregate both the features generating either positive or negative price variation. The overall accuracy of the classification is negatively impacted by the overlap of the features from the two classes.

In the second case, the classifier has to consider only the observations located on one side of the hyperplane equation, without taking into account the false negatives.

The accuracy of the binary classifier can be further improved by considering the positive variation of price as price(t+1) – price(t) > EPS.