The SVR introduces the concept of error insensitive zone and insensitive error, ε. The insensitive zone defines a range of values around the predictive values, y(x). The penalization component C does not affect the data point {x_i,y_i} that belongs to the insensitive zone [8:14].

The following diagram illustrates the concept of an error insensitive zone using a single variable feature x and an output y. In the case of a single variable feature, the error insensitive zone is a band of width 2ε (ε is known as the insensitive error). The insensitive error plays a similar role to the margin in the SVC.

The visualization of the support vector regression and insensitive error

For the mathematically inclined, the maximization of the margin for nonlinear models introduces a pair of slack variables. As you may remember, the C-support vector classifiers use a single slack variable. The preceding diagram illustrates the minimization formula.

Note

M9: The ε-SVR formulation is defined as:

Here, ε is the insensitive error function.

M10: The ε-SVR regression equation is given by:

Let's reuse the SVM class to evaluate the capability of the SVR, compared to the linear regression (refer to the Ordinary least squares regression section in Chapter 6, Regression and Regularization).

This test consists of reusing the example on single-variate linear regression (refer to the One-variate linear regression section in Chapter 6, Regression and Regularization). The purpose is to compare the output of the linear regression with the output of the SVR for predicting the value of a stock price or an index. We select the S&P 500 exchange traded fund, SPY, which is a proxy for the S&P 500 index.

The model consists of the following:

The implementation follows a familiar pattern:

The code will be as follows:

val path = "resources/data/chap8/SPY.csv"
val C = 12
val GAMMA = 0.3
val EPSILON = 2.5

val config = SVMConfig(new SVRFormulation(C, EPSILON), 
    new RbfKernel(GAMMA)) //45
for {
  price <-  DataSource(path, false, true, 1) get close
  (xt, y) <- getLabeledData(price.size)  //46
  linRg <- SingleLinearRegression[Double](price, y) //47
  svr <- SVM[Double](config, xt, price)
} yield {
  collect(svr, linRg, price)
}

The formulation in the configuration has the SVRFormulation type (line 45). The DataSource class extracts the price of the SPY ETF. The getLabeledData method generates the xt input features and the y labels (or expected values) (line 46):

type LabeledData = (Vector[DblArray], DblVector)
def getLabeledData(numObs: Int): Try[LabeledData ] = Try {
    val y = Vector.tabulate(numObs)(_.toDouble)
    val xt = Vector.tabulate(numObs)(Array[Double](_))
    (xt, y)
}

The single variate linear regression, SingleLinearRegression, is instantiated using the price input and y labels as inputs (line 47).

Finally, the collect method executes the two pfSvr and pfLinr regression partial functions:

def collect(svr: SVM[Double], 
   linr: SingleLinearRegression[Double], price: DblVector){
  
  val pfSvr = svr |>
  val pfLinr = linr |>
  for {
    if( pfSvr.isDefinedAt(n.toDouble))
    x <- pfSvr(n.toDouble) 
    if( pfLin.isDefinedAt(n))
    y <- pfLinr(n)
  } yield  {  ... }
}

The results are displayed in the following graph, which are generated using the JFreeChart library. The code to plot the data is omitted because it is not essential to the understanding of the application.

A comparative plot of linear regression and SVR

The support vector regression provides a more accurate prediction than the linear regression model. You can also observe that the L₂ regularization term of the SVR penalizes the data points (the SPY price) with a high deviation from the mean of the price. A lower value of C will increase the L₂-norm penalty factor as λ =1/C.

There is no need to compare SVR with the logistic regression, as the logistic regression is a classifier. However, the SVM is related to the logistic regression; the hinge loss in the SVM is similar to the loss in the logistic regression [8:15].