Constructing cv::ml::TrainData

The cv::ml::TrainData class allows you to package up your data, along with some instructions about how it is to interpreted and used in training. In practice, this additional information is extremely useful. Here is the create() method used to generate a new cv::ml::TrainData object.

// Construct training data from the specified matrix of data points and responses.
// It's possible to use a subset of features (a.k.a. variables) and/or subset of
// samples; it's possible to assign weights to individual samples.
//
static cv::Ptr<cv::ml::TrainData> cv::ml::TrainData::create(
  cv::InputArray samples,                       // Array of samples (CV_32F)
  int            layout,                        // row/col (see ml::SampleTypes)
  cv::InputArray responses,                     // Float array of responses
  cv::Inputarray varIdx        = cv::noArray(), // Specifies training variables
  cv::InputArray sampleIdx     = cv::noArray(), // Specifies training samples
  cv::InputArray sampleWeights = cv::noArray(), // Optional sample wts (CV_32F)
  cv::InputArray varType       = cv::noArray()  // Optional, types for each
                                                //  input and output
                                                //  variable (CV_8U)
);

This method constructs training data from the preallocated arrays of training samples and the associated responses. The matrix of samples must be of type CV_32FC1 (32-bit, floating-point, and single-channel). Though the cv::Mat class is clearly capable of representing multichannel images, the machine learning algorithms take only a single channel—that is, just a two-dimensional array of numbers. Typically, this array is organized as rows of data points, where each “point” is represented as a vector of features. Hence, the columns contain the individual features for each data point and the data points are stacked to yield the 2D single-channel training matrix. To belabor the topic: the typical data matrix is thus composed of (rows, columns) = (data points, features).

Some of the algorithms can handle transposed matrices directly. The parameter layout specifies how the data is stored:

layout = cv::ml::ROW_SAMPLE

Means that the feature vectors are stored as rows (this is the most common layout)

layout = cv::ml::COL_SAMPLE

Means that the feature vectors are stored as columns

You may well ask: What if my training data is not floating-point numbers but instead is letters of the alphabet or integers representing musical notes or names of plants? The answer is: Fine, just turn them into unique 32-bit floating-point numbers when you fill the cv::Mat. If you have letters as features or labels, you can cast the ASCII character to floats when filling the data array. The same applies to integers. As long as the conversion is unique, things should work out fine. Remember, however, that some routines are sensitive to widely differing variances among features. It’s generally best to normalize the variance of features, as we saw in the previous section. With the exception of the tree-based algorithms (decision trees, random trees, and boosting) that support both categorical and ordered input variables, all other OpenCV ML algorithms work only with ordered inputs. A popular technique for making ordered-input algorithms work with categorical data is to represent them in “1-radix” or “1-hot” notation; for example, if the input variable color may have seven different values, then it may be replaced by seven binary variables, where one and only one of the variables may be set to 1.³

The responses parameter will contain either categorical labels such as poisonous or nonpoisonous, in the case of mushroom identification, or are regression values (numbers) such as body temperatures taken with a thermometer. The response values, or “labels,” are usually a one-dimensional vector with one value per data point. One important exception is neural networks, which can have a vector of responses for each data point. For categorical responses, the response value type must be an integer (CV_32SC1); for regression problems, the response should be of 32-bit floating-point type (CV_32FC1). In the special case of neural networks, as alluded to before, it is common to put a little twist on this, and actually perform categorization using a regression framework. In this case, the 1-hot encoding mentioned earlier is used to represent the various categories and floating-point output is used for all of the multiple outputs. In this case the network is, in essence, being trained to regress to something like the probability that the input is in each category.

Recall, however, that some algorithms can deal only with classification problems and others only with regression, while still others can handle both. In this last case, the type of output variable is passed either as a separate parameter or through the varType vector. This vector can be either a single column or a single row, and must be of type CV_8UC1 or cv::S8C1. The number of entries in varType is equal to the number of input variables () plus the number of responses (typically one).⁴ The first entries will tell the algorithm the type of the corresponding input feature, while the remainder indicate the types of the output. Each entry in varType should be set to one of the following values:

cv::ml::VAR_CATEGORICAL

Means that the output values are discrete class labels

cv::ml::VAR_ORDERED (= cv::ml::VAR_NUMERICAL)

Means that the output values are ordered; that is, different values can be compared as numbers and so this is a regression problem

Many models in the ML library may be trained on a selected feature subset and/or on a selected sample subset of the training set. To make this easier for the user, the cv::ml::TrainData::create() method includes the vectors varIdx and sampleIdx as parameters. The varIdx vector can be used to identify specific variables (features) of interest, while sampleIdx can identify specific data points of interest. Either of these may simply be omitted or set to cv::noArray() (the default value) to indicate that you would like to use “all of the features” or “all of the points.” Both vectors are either provided as lists of zero-based indices or as masks of active variables/samples, where a nonzero value signifies active. In the former case, the vector must be of type CV_32SC1 and may have any length. In the latter case, the array must be of type CV_8UC1 and must have the same length as the number of features or samples (as appropriate). The parameter sampleIdx is particularly helpful when you’ve read in a chunk of data and want to use some of it for training and some of it for testing without having to first break it into two different vectors.

Constructing cv::ml::TrainData from stored data

Often, you will have data already saved on disk. If this data is in CSV (comma-separated value) format, or you can put it into this format, you can create a new cv::ml::TrainData object from that CSV file using cv::ml::TrainData::loadFromCSV().

// Load training data from CSV file; part of each row may be treated as the
// scalar or vector responses; the rest are input values.
//
static cv::Ptr<cv::ml::TrainData> cv::ml::TrainData::loadFromCSV(
   const String& filename,                    // Input file name
   int           headerLineCount,             // Ignore this many lines
   int           responseStartIdx = -1,       // Idx of first out var (-1=last)
   int           responseEndIdx   = -1,       // Idx of last out var plus one
   String&       varTypeSpec      = String(), // Optional, specifies var types
   char          delimiter        = ',',      // Char used to separate values
   char          missch           = '?'       // Used for missing data in CSV
);

The CSV reader skips the first headerLineCount lines and then reads the data. The data is read row by row⁵ and individual features are separated based on commas. If some other separator is used in the CSV file, the delimeter argument may be used to replace the default comma (e.g., by a space or semicolon). Often, the responses will be found in the leftmost or the rightmost column, but the user may specify any column (or a range of columns) as necessary. The responses will be drawn from the interval [responseStartIdx, responseEndIdx), inclusive of responseStartIdx but exclusive of responseEndIdx. The variable types, if required, are specified via single compact text string, vatTypeSpec. For example:

"ord[0-9,11]cat[10]"

means that the data has 12 columns, the first 10 columns contain ordered values, then there is column of categorical values, and then there is yet another column of ordered values.

If you do not provide a variable type specification, then the reader tries to “do the right thing” by following a few simple rules. It considers input variables to be ordered (numerical) unless they clearly contain non-numerical values (e.g., "dog", "cat"), in which case they are made categorical. If there is only one output variable, then it will follow pretty much the same rule,⁶ but if there are multiple, then they are always considered ordered.

It is also possible to specify a special character, using the missch argument, to be used for missing measurements. However, it is important to know that some of the algorithms cannot handle such missing values. In such cases missing points should be interpolated or otherwise handled by the user before training or the corrupted records should be rejected in advance.⁷

Secret sauce and cv::ml::TrainDataImpl

Were you to go to the source code and look at the class definition for cv::ml::TrainData, you would see that it is full of pure virtual functions. In fact, it is just an interface, from which you can derive and create your own training data containers. This fact immediately leads to two obvious questions: why would you want to do this, and what exactly is going on inside of cv::ml::TrainData::create() if cv::ml::TrainData is a virtual class type?

As for the why, training data can be very complex in real-life situations and the data itself can be very large. In many cases it is necessary to implement more sophisticated strategies for managing the data and for storing it. For these reasons, you might want to implement your own training data container that, for example, uses a database for the management of the bulk of the available data. Using the cv::ml::TrainData interface, you can implement your own data container and the available algorithms will run on that data transparently.

For the second question, the how, the answer is that the cv::ml::TrainData::create() method actually creates an object of a different class than it appears to. There is a class called cv::ml::TrainDataImpl that is essentially the default implementation of a data container. This object manages the data just the way you would expect—in the form of a few arrays inside that hold the various things you think you have put in there.

In fact, the existence of this class will be largely invisible to you unless (until) you start looking at the library source code directly. Of course, if you do find yourself wanting to build your own cv::ml::TrainData–derived container class, it will be very useful to look at the implementation of cv::ml::TrainDataImpl in .../opencv/modules/ml/src/data.cpp.

Splitting training data

In practice, when you are training a machine learning system, you don’t want to use all of the data you have to train the algorithm. You will need to hold some back to test the algorithm when it is done. If you don’t do this, you will have no way of estimating how your trained system will behave when presented with novel data. By default, when you construct a new instance of TrainData, it’s all considered available to be used for training data, and none of it is held back for such testing. Using cv::ml::TrainData::setTrainTestSplit(), you can split the data into a training and a test part, and just use the training part for training your model. Using just this training data is the automatic behavior of cv::ml::StatModel::train(), assuming you have marked what data you want it to use.

// Splits the training data into the training and test parts
//
void cv::ml::TrainData::setTrainTestSplit(
  int    count,
  bool   shuffle = true
);

void cv::ml::TrainData::setTrainTestSplitRatio(
  double ratio,
  bool   shuffle = true
);

void cv::ml::TrainData::shuffleTrainTest();

The three members of cv::ml::TrainData that will help you out with this are: setTrainTestSplit(), setTrainTestSplitRatio(), and shuffleTrainTest(). The first takes a count argument that specifies how many of the vectors in the data set should be labeled as training data (with the remainder being test data). Similarly the second function does the same thing, but allows you to specify the ratio of points (e.g., 0.90 = 90%) that will be labeled as training data. Finally, the third “shuffle” method will randomly assign the train and test vectors (while keeping the number of each fixed). Either of the first two methods supports a shuffle argument. If true, then the test and train labels will be assigned randomly; otherwise, the train samples will start from the beginning and the test samples will be those vectors thereafter.

Note that internally, the default implementation IMPL has three separate indices that do similar things: the sample index, the train index, and the test index. Each is a list of indices into the overall array of samples in the container that indicates which samples are to be used in a particular context. The sample index is an array listing all samples that will be used. The train index and test index are similar, but list which samples are for training and which are for testing. As implemented, these three indices have a relationship that some might find unintuitive.

If the train index is defined, it should always be the case that the test index is defined. This is the natural result of the use of the functions just described to create these internal indices. If either (both) is defined, then its behavior will always define how train() responds; this is regardless of anything that might be in the sample index. Only when these two indices are undefined will the sample index be used. In that case, all data indicated by the sample index will be assumed available for training, and no data will be marked as test data.

Accessing cv::ml::TrainData

Once the training data is constructed, it’s possible to retrieve its parts, with or without preprocessing, using the methods described next. The function cv::ml::TrainData::getTrainSamples() retrieves a matrix only of the training data.

// Retrieve only the active training data into a cv::Mat array.
//
cv::Mat cv::ml::TrainData::getTrainSamples(
  int  layout          = ROW_SAMPLE,
  bool compressSamples = true,
  bool compressVars    = true
) const;

When compressSamples or compressVars are true, the method will retain only rows or columns set by sampleIdx and varIdx, respectively (typically at construction time). The method also transposes the data if the desired layout is different from the original one. If the sample index or the train index is defined, then only the indicated samples will be returned. Recall, however, that, if both are defined, it will be the train index that determines what is returned.

Similarly, the cv::ml::TrainData::getTrainResponses() method extracts only the active response vector elements.

// Return the train responses (for the samples selected using sampleIdx).
//
cv::Mat cv::ml::TrainData::getTrainResponses() const;

As with cv::ml::TrainData::getTrainSamples(), if the sample index or the train index is defined, then only the indicated samples will be returned. Recall, however, that, if both are defined, it will be the train index that determines what is returned.

Similarly, there are two functions—cv::ml::TrainData::getTestSamples() and cv::ml::TrainData::getTestResponses()—that return the analogous arrays constructed from only the test samples. In this case, however, if the test index is not defined, then an empty array will be returned.

Finally, there are accessors that will simply tell you how many of various kinds of samples are in the data container. We list them here.

int getNTrainSamples() const;  // Number of samples indicated by train idx
                               //  or total samples if samples idx is not defined

int getNTestSamples()  const;  // Number of samples indicated by test idx
                               //  or zero test idx is not defined

int getNSamples()      const;  // Number of samples indicated by samples idx
                               //  or total samples if samples idx is not defined

int getNVars()         const;  // Number of features indicated by variable idx
                               //  or total samples if variable idx is not defined

int getNAllVars()      const;  // Number of features total

The naïve/normal Bayes classifier and cv::ml::NormalBayesClassifier

The following is the class definition for the normal Bayes classifier. Note that the class name cv::ml::NormalBayesClassifier is actually another layer of interface definition, while cv::ml::NormalBayesClassifierImpl is the name of the actual class that implements the normal Bayes classifier. For convenience, this definition lists some important inherited methods as comments.

// Somewhere above...
// namespace cv {
//   namespace ml {
//
class NormaBayesClassifierImpl : public NormaBayesClassifier {
                                      // cv::ml::NormaBayesClassifier is derived
                                      // from cv::ml::StatModel
public:

  ...

  float predictProb(
    InputArray   inputs,
    OutputArray  outputs,
    OutputArray  outputProbs,
    int          flags      = 0
  );

  ...


  // From class NormaBayesClassifier
  //
  // Ptr<NormaBayesClassifier> NormaBayesClassifier::create(); // constructor


};

The training method for the normal Bayes classifier, inherited from cv::ml::StatModel, is:

bool cv::ml::NormalBayesClassifier::train(
  const Ptr<cv::ml::TrainData>& trainData,   // your data
  int flags = 0                              // 0=new data or UPDATE_MODEL=add
);

The flags parameter may be 0 or include the cv::ml::StatModel::UPDATE_MODEL flag, which means that the model needs to be updated using the additional training data rather than retrained from scratch.

The cv::NormalBayesClassifier implements the inherited predict() interface described in cv::ml::StatModel, which computes and returns the most probable class for its input vectors. If more than one input data vector (row) is provided in the samples matrix, the predictions are returned in corresponding rows of the results vector. If there is only a single input in samples, then the resulting prediction is also returned as a float value by the predict() method and the results array may be set to cv::noArray().

float cv::ml::NormalBayesClassifier::predict(
  cv::InputArray  samples,                  // input samples, float matrix
  cv::OutputArray results = cv::noArray(),  // optional output results matrix
  int             flags   = 0               // (model-dependent)
) const;

Alternatively, the normal Bayes classifier also offers the method predictProb(). This method takes the same arguments as cv::ml::NormalBayesClassifier::predict(), but also the arrray resultProbs. This is a floating-point matrix of number_of_samples × number_of_classes size, where the computed probabilities (that the corresponding samples belong to the particular classes) will be stored.¹³ The format for this prediction method is:

float cv::ml::NormalBayesClassifier::predictProb( // prob if single sample
  InputArray  samples,                            // one sample per row
  OutputArray results,                            // predictions, one per row
  OutputArray resultProbs,                        // row=sample, column=class
  int         flags = 0                           // 0 or StatModel::RAW_OUTPUT
) const;

Though the naïve Bayes classifier is extremely useful for small data sets, it does not generally perform well when the data has a great degree of structure. With this in mind, we move next to a discussion of tree-based classifiers, which can dramatically outperform something as simple as the naïve Bayes classifier, particularly when sufficient data is present.

Regression impurity

For regression or function fitting, the equation for node impurity is simply the square of the difference in value between the node value y and the data value x. We want to minimize:

Classification impurity

For classification, decision trees often use one of three methods: entropy impurity, Gini impurity, or misclassification impurity. For these methods, we use the notation to denote the fraction of patterns at node N that are in class . Each of these impurities has slightly different effects on the splitting decision. Gini is the most commonly used, but all the algorithms attempt to minimize the impurity at a node. Figure 21-3 graphs the impurity measures that we want to minimize. In practice, it is best to just try each impurity to determine on a validation set which one works best.

Figure 21-3. Decision tree impurity measures

Entropy impurity:

Gini impurity:

Misclassification impurity:

Decision trees are perhaps the most widely used classification technology. This is due to their simplicity of implementation, ease of interpretation of results, flexibility with different data types (categorical, numerical, unnormalized, and mixes thereof), ability to handle missing data through surrogate splits, and natural way of assigning importance to the data features by order of splitting. Decision trees form the basis of other algorithms such as boosting and random trees, which we will discuss shortly.

OpenCV implementation

The following is the abbreviated declaration of cv::ml::DTrees. The train() methods are just derived from the base class; most of what we need in the definition is how the parameters to the model are set and loaded.

// Somewhere above...
// namespace cv {
//   namespace ml {
//
class DTreesImpl : public Dtrees {    // cv::ml::DTrees is derived
                                      // from cv::ml::StatModel
public:

  // (Inherited from cv::ml::DTrees)
  //
  //enum Flags {
  //  PREDICT_AUTO     = 0,
  //  PREDICT_SUM      = (1<<8),
  //  PREDICT_MAX_VOTE = (2<<8),
  //  PREDICT_MASK     = (3<<8)
  //};

  int     getCVFolds() const;                 // get num cross validation folds
  int     getMaxCategories() const;           // get max number of categories
  int     getMaxDepth() const;                // get max tree depth
  int     getMinSampleCount() const;          // get min sample count
  Mat     getPriors() const;                  // get priors for categories
  float   getRegressionAccuracy() const;      // get required regression acc.
  bool    getTruncatePrunedTree() const;      // get to truncate pruned trees
  bool    getUse1SERule() const;              // get for use 1SE rule in pruning
  bool    getUseSurrogates() const;           // get to use surrogates

  void    setCVFolds( int val );              // set num cross validation folds
  void    setMaxCategories( int val );        // set max number of categories
  void    setMaxDepth( int val );             // set max tree depth
  void    setMinSampleCount( int val );       // set min sample count
  void    setPriors( const cv::Mat &val );    // set priors for categories
  void    setRegressionAccuracy( float val ); // set required regression acc.
  void    setTruncatePrunedTree( bool val );  // set to truncate pruned trees
  void    setUse1SERule( bool val );          // set for use 1SE rule in pruning
  void    setUseSurrogates( bool val );       // set to use surrogates

  ...

  // Experts can use these, but non-experts can ignore them.
  //
  const std::vector<Node>&  getNodes()   const;
  const std::vector<int>&   getRoots()   const;
  const std::vector<Split>& getSplits()  const;
  const std::vector<int>&   getSubsets() const;

  ...

  // From class DTrees
  //
  //Ptr<DTrees> DTrees::create(); // The algorithm "constructor",
  //                              //  returns Ptr<DTreesImpl>

};

First of all, you might have noticed that the name is plural: DTrees. This is because in computer vision it’s not standalone decision trees that are primarily used, but rather ensembles of decision trees—that is, collections of decision trees that produce some joint decision. Two such popular ensembles, implemented in OpenCV and discussed later in this chapter, are RTrees (random trees) and Boost (boosting). Both share a lot of internal machinery and so DTrees is a sort of base class that, in our current case, can be viewed as an ensemble consisting of a single tree.

The second thing you might have noticed is that again, as we saw with the data container cv::ml::TrainData, these prototypes are mostly pure virtual. This is essentially the same thing again. There is actually a hidden class called cv::ml::DTreesImpl that is derived from cv::ml::DTrees and that contains all of the default implementation. The bottom line is that you can effectively ignore the fact that those functions are pure virtual in the preceding class definition. If, however, at some point you would like to go and look at the default implementation, you will need this information in order to find the member functions you want in .../modules/ml/src/tree.cpp.

Once you have constructed a cv::ml::DTrees object using cv::ml::DTrees::create(), you will need to configure the various runtime parameters. You can do this one of two ways. You will need to construct a structure that contains the needed parameters to configure the tree. This structure is called cv::ml::TreeParams. The salient parts of its definition are given next.

You will notice that you can create the structure either with the form of its constructor that has an argument for every component, or you can just create it using the default constructor—in which case everything will be set to default values—and then use individual accessors to set the values you want to customize.

Table 21-1 contains a brief description of the components of cv::ml::TreeParams(), their default values, and their meanings.

Two of these arguments warrant a little further investigation. maxCategories limits the number of categorical values before which the decision tree will precluster those categories so that it will have to test no more than possible value subsets.¹⁶ Those variables that have more categories than maxCategories will have their category values clustered down to maxCategories possible values. In this way, decision trees will have to test no more than maxCategories levels at a time, which results in considering no more than possible decision subsets for each categorical input. This parameter, when set to a low value, reduces computation but at the cost of accuracy.

The last parameter, priors, sets the relative weight that you give to misclassification. That is, if we build a two-class classifier and if the weight of the first output class is 1 and the weight of the second output class is 10, then each mistake in predicting the second class is equivalent to making 10 mistakes in predicting the first class. In the example we will look at momentarily, we use edible and poisonous mushrooms. In this context, it makes sense to “punish” mistaking a poisonous mushroom for an edible one 10 times more than mistaking an edible mushroom for a poisonous one. The parameter priors is an array of floats with the same number of elements as there are classes. The assigned values are in the same order as the classes themselves.

The train() method is derived directly from cv::ml::StatModel:

// Work directly with decision trees:
//
bool cv::ml::DTrees::train(
const cv::Ptr<cv::ml::TrainData>& trainData,   // your data
int                               flags    = 0 // use UPDATE_MODEL to add data
);

In the train() method, we have the floating-point trainData matrix. With decision trees, you can set the layout to cv::COL_SAMPLE when constructing trainData if you want to arrange your data into columns instead of the usual rows, which is the most efficient layout for this algorithm. Example 21-1 details the creation and training of a decision tree.

The function for prediction with a decision tree is the same as for its base class, cv::ml::StatModel:

float cv::ml::DTrees::predict(
  cv::InputArray  samples,
  cv::OutputArray results = cv::noArray(),
  int             flags   = 0
) const;

Here, samples is a floating-point matrix, with one row per sample. In the case of a single input, the return value will be enough and results can be set to cv::noArray(). In the case of multiple vectors to be evaluated, the output results will contain a prediction for each input vector. Finally, flags specifies various possible options. For example, cv::ml::StatModel::PREPROCESSED_INPUT indicates that the values of each categorical variable j are normalized to the range 0..N_j – 1, where N_j is the number of categories for jth variable. For example, if some variable may take just two values, A and B, after normalization A is converted to 0 and B to 1. This is mainly used in ensembles of trees to speed up prediction. Normalizing data to fit within the (0, 1) interval is simply a computational speedup because the algorithm then knows the bounds in which data may fluctuate. Such normalization has no effect on accuracy. This method returns the predicted value, normalized (when flags includes cv::ml::StatModel::RAW_OUTPUT) or converted to the original label space (when flags does not include the RAW_OUTPUT flag).

Most users will only train and use the decision trees, but advanced or research users may sometimes wish to examine and/or modify the tree nodes or the splitting criteria. As stated in the beginning of this section, the information for how to do this is in the ML documentation online at http://docs.opencv.org. The sections of interest for such advanced analysis are the class structure cv::ml::DTrees, the node structure cv::ml::DTrees::Node, and its contained split structure cv::ml::DTrees::Split.

Decision tree usage

We will now explore the details by looking at a specific example. Consider a program whose purpose is to learn to identify poisonous mushrooms. There is a public data set called agaricus-lepiota.data that contains information about some 8,000 different kinds of mushrooms. It lists many features that might distinguish a mushroom visually, such as the color of the cap, the size and spacing of the gills, as well as—and this is very important—whether or not that type of mushroom is poisonous.¹⁷ The data file is in CSV format and consists of a label 'p' or 'e' (denoting poisonous or edible, respectively) followed by 22 categorical attributes, each represented by a single letter. It should also be noted that the file contains examples in which certain data is missing (i.e., one or more of the attributes is unknown for that particular type of mushroom). In this case, the entry is a '?' for that feature.

Let’s take the time to look at this program in detail, which will use binary decision trees to learn to recognize poisonous from edible mushrooms based on their various visible attributes (Example 21-1).¹⁸

#include <opencv2/opencv.hpp>
#include <stdio.h>
#include <iostream>

using namespace std;
using namespace cv;

int main( int argc, char* argv[] ) {

    // If the caller gave a filename, great. Otherwise, use a default.
    //
    const char* csv_file_name = argc >= 2
    ? argv[1]
    : "agaricus-lepiota.data";

    cout <<"OpenCV Version: " <<CV_VERSION <<endl;

    // Read in the CSV file that we were given.
    //
    cv::Ptr<cv::ml::TrainData> data_set = cv::ml::TrainData::loadFromCSV(
      csv_file_name,   // Input file name
      0,               // Header lines (ignore this many)
      0,               // Responses are (start) at thie column
      1,               // Inputs start at this column
      "cat[0-22]"      // All 23 columns are categorical
    );                 // Use defaults for delimeter (',') and missch ('?')

    // Verify that we read in what we think.
    //
    int n_samples = data_set->getNSamples();
    if( n_samples == 0 ) {
        cerr <<"Could not read file: " <<csv_file_name <<endl;
        exit( -1 );
    } else {
        cout <<"Read " <<n_samples <<" samples from " <<csv_file_name <<endl;
    }

    // Split the data, so that 90% is train data
    //
    data_set->setTrainTestSplitRatio( 0.90, false );
    int n_train_samples = data_set->getNTrainSamples();
    int n_test_samples  = data_set->getNTestSamples();

    cout <<"Found " <<n_train_samples <<" Train Samples, and "
    <<n_test_samples <<" Test Samples" <<endl;

    // Create a DTrees classifier.
    //
    cv::Ptr<cv::ml::RTrees> dtree = cv::ml::RTrees::create();

    // set parameters
    //
    // These are the parameters from the old mushrooms.cpp code

    // Set up priors to penalize "poisonous" 10x as much as "edible"
    //
    float _priors[] = { 1.0, 10.0 };
    cv::Mat priors( 1, 2, CV_32F, _priors );

    dtree->setMaxDepth( 8 );
    dtree->setMinSampleCount( 10 );
    dtree->setRegressionAccuracy( 0.01f );
    dtree->setUseSurrogates( false /* true */ );
    dtree->setMaxCategories( 15 );
    dtree->setCVFolds( 0 /*10*/ );  // nonzero causes core dump
    dtree->setUse1SERule( true );
    dtree->setTruncatePrunedTree( true );
    //dtree->setPriors( priors );
    dtree->setPriors( cv::Mat() ); // ignore priors for now...

    // Now train the model
    // NB: we are only using the "train" part of the data set
    //
    dtree->train( data_set );

    // Having successfully trained the data, we should be able
    // to calculate the error on both the training data, as well
    // as the test data that we held out.
    //
    cv::Mat results;
    float train_performance = dtree->calcError(
                                               data_set,
                                               false,      // use train data
                                               results //cv::noArray()
                                               );
    std::vector<cv::String> names;
    data_set->getNames(names);
    Mat flags = data_set->getVarSymbolFlags();

    // Compute some statistics on our own:
    //
    {
        cv::Mat expected_responses = data_set->getResponses();
        int good=0, bad=0, total=0;

        for( int i=0; i<data_set->getNTrainSamples(); ++i ) {
            float received = results.at<float>(i,0);
            float expected = expected_responses.at<float>(i,0);
            cv::String r_str = names[(int)received];
            cv::String e_str = names[(int)expected];

            cout <<"Expected: " <<e_str <<", got: " <<r_str <<endl;

            if( received==expected ) good++; else bad++; total++;
        }
        cout <<"Correct answers:   " <<(float(good)/total) <<"%" <<endl;
        cout <<"Incorrect answers: " <<(float(bad)/total)  <<"%" <<endl;
    }

    float test_performance  = dtree->calcError(
                                               data_set,
                                               true,       // use test data
                                               results //cv::noArray()
                                               );

    cout <<"Performance on training data: " <<train_performance <<"%" <<endl;
    cout <<"Performance on test data:     " <<test_performance  <<"%" <<endl;

    return 0;
}

We start out by parsing the command line for a single argument, the CSV file to read; if there is no such file, we read the mushroom file by default. We print the OpenCV version number,¹⁹ and then continue on to parse the CSV file. This file has no header, but the results are in the first column, so it is important to specify that. Finally, all of the inputs are categorical, so we state that explicitly. Once we have read the CSV file, we state how many samples were read, which is a good way to verify that the reading went well and that you read the file you thought you were reading.

Next, we set the train-test split. In this case, we do so with setTrainTestRatio(), and so we specify what fraction we would like to be training data; in this case it is 90% (0.90). Also note that the default behavior of the split is to also shuffle the data. If we do not want to do that shuffling, we need to pass false as the second argument. Once this split is done, we print out how many training samples there are and how many test samples.

Once the data is all prepared, we can go on and create the cv::ml::DTrees object that we will be training. This object is configured with a series of calls to its various set*() methods. Notably, we pass an array of values to setPriors(). This allows us to set relative weight of missing a poisonous mushroom as opposed to incorrectly marking an edible mushroom as poisonous. The reason the 1.0 comes first and the 10.0 comes after is because e comes before p in the alphabet (and thus, once converted to ASCII, then to a floating-point number, it comes first numerically).

In this example, we simply train DTrees and then use it to predict results on some test data. In a more realistic application, the decision tree may also be saved to disk via save() and loaded via load() (see the following). In this way, it is possible to train a classifier and then distribute the trained classifier in your code, without having to distribute the data (or make your users retrain every time!) The following code shows how to save and to load a tree file called tree.xml.

// To save your trained classifier to disk:
//
dtree->save("tree.xml","MyTree");

// To load a trained classifier from disk:
//
dtree->load("tree.xml","MyTree");

// You can also clear an existing trained classifier.
//
dtree->clear();

Decision tree results

By tinkering with the previous code and experimenting with various parameters, we can learn several things about edible or poisonous mushrooms from the agaricus-lepiota.data file. If we just train a decision tree without pruning and without priors, so that it learns the data perfectly, we might get the tree shown in Figure 21-4. Although the full decision tree may learn the training set of data perfectly, remember the lessons from the section “Diagnosing Machine Learning Problems” in Chapter 20 about variance/overfitting. What’s happened in Figure 21-4 is that the data has been memorized along with its mistakes and noise. Such a tree unlikely to perform well on real data. For this reason, OpenCV decision trees (and CART type trees generally) typically include the additional step of penalizing complex trees and pruning them back until complexity is in balance with performance.²⁰

Figure 21-5 shows a pruned tree that will still do quite well (but not perfectly) on the training set but will probably perform better on real data because it has a better balance between bias and variance. However, the classifier shown has a serious shortcoming: although it performs well on the data, it now labels poisonous mushrooms as edible 1.23% of the time.

Figure 21-4. Full decision tree for poisonous (p) or edible (e) mushrooms: this tree was built out to full complexity for 0% error on the training set and so would probably suffer from variance problems on test or real data

Figure 21-5. Pruned decision tree for poisonous (p) and edible (e) mushrooms. Despite being pruned, this tree shows low error on the training set and would likely work well on real data

As you might imagine, there are big advantages to a classifier that may label many edible mushrooms as poisonous but which nevertheless does not invite us to eat a poisonous mushroom! As we saw previously, we can create such a classifier by intentionally biasing (or as adding a cost to) the classifier and/or the data. This is what we did in Example 21-1, we added a higher cost for misclassifying poisonous mushrooms than for misclassifying edible mushrooms. By adjusting the priors vector, we imposed a cost into the classifier and, as a result, changed the weighting of how much a “bad” data point counts versus a “good” one. Alternatively, if one did not, or could not, modify the classifier code to change the prior, one can equivalently impose additional cost by duplicating (or resampling from) “bad” data. Duplicating “bad” data points implicitly gives a higher weight to the “bad” data, a technique that can work with almost any classifier.

Figure 21-6 shows a tree where a 10× bias was imposed against poisonous mushrooms. This tree makes no mistakes on poisonous mushrooms at a cost of many more mistakes on edible mushrooms, a case of “better safe than sorry.” Confusion matrices for the (pruned) unbiased and biased trees are shown in Figure 21-7.

Figure 21-6. An edible mushroom decision tree with 10× bias against misidentification of poisonous mushrooms as edible; note that the lower-right rectangle, though containing a vast majority of edible mushrooms, does not contain a 10× majority and so would be classified as inedible

Figure 21-7. Confusion matrices for (pruned) edible mushroom decision trees: the unbiased tree yields better overall performance (top panel) but sometimes misclassifies poisonous mushrooms as edible; the biased tree does not perform as well overall (lower panel) but never misclassifies poisonous mushrooms

AdaBoost

Boosting algorithms are used to train weak classifiers , . These classifiers are generally very simple individually. In most cases these classifiers are decision trees with only one split (called decision stumps) or at most a few levels of splits (perhaps up to three). Each of the classifiers is assigned a weighted vote in the final decision-making process for the resulting final strong classifier. We use a labeled data set of input feature vectors , each with scalar label y_i (where indexes the data points). For AdaBoost the label is binary, , though it can be any floating-point number in other algorithms. We also initialize a data point weighting distribution ; this tells the algorithm how much misclassifying a data point will “cost.” The key feature of boosting is that, as the algorithm progresses, this cost will evolve so that weak classifiers trained later will focus on the data points that the earlier trained weak classifiers tended to do poorly on. The algorithm is as follows:

1. .

2. For :

   1. Find the classifier that minimizes the weighted error:

      , where (for as long as ; else quit).

   2. Set the “voting weight” , where is the argmin error from Step 2a.

   3. Update the data point weights:

      

   Here, normalizes the equation over all data points , so that for all w.

Note that, in Step 2a, if we can’t find a classifier with less than a 50% error rate then we quit; we probably need better features.

When the training algorithm just described is finished, the final strong classifier takes a new input vector and classifies it using a weighted sum over the learned weak classifiers :

Here, the sign function converts anything positive into a 1 and anything negative into a –1 (zero remains 0). For performance reasons the leaf values of the just trained ith decision tree are scaled by and then H(x) reduces to the sign of the sum of weak classifier responses on x.

Boosting code

The code for boosting is similar to the code for decision trees, and the cv::ml::Boost class is  derived from cv::ml::DTrees with a few extra control parameters. As we have seen elsewhere in the library, when you call cv::ml::Boost::create(), you will get back an object of type cv::Ptr<cv::ml::Boost>, but this object will actually be a pointer to an object of the (generally invisible) cv::ml::BoostImpl class type.

// Somewhere above..
// namespace cv {
// namespace ml {
//
class BoostImpl : public Boost {  // cv::ml::Boost is derived from cv::ml::DTrees

public:

  // types of boosting
  // (Inherited from cv::ml::Boost)
  //
  //enum Types {
  //  DISCRETE = 0,
  //  REAL     = 1,
  //  LOGIT    = 2,
  //  GENTLE   = 3
  //};

  // get/set one of boosting types:
  //
  int    getBoostType() const;        // get type: DISCRETE, REAL, LOGIT, GENTLE
  int    getWeakCount() const;        // get the number of weak classifiers
  double getWeightTrimRate() const;   // get the trimming rate, see text

  void setBoostType(int val);         // get type: DISCRETE, REAL, LOGIT, GENTLE
  void setWeakCount(int val);         // get the number of weak classifiers
  void setWeightTrimRate(double val); // get the trimming rate, see text

  ...

  // from class Boost
  //
  //static Ptr<Boost> create(); // The algorithm "constructor",
  //                            //  returns Ptr<BoostImpl>

};

The member setWeakCount() sets the number of weak classifiers that will be used to form the final strong classifier. The default value for this number is 100.

The number of weak classifiers is distinct from the maximum complexity that is allowed to each of the individual classifiers. The latter is controlled by setMaxDepth(), which sets the maximum number of layers that an individual weak classifier can have. As mentioned earlier, a value of 1 is common, in which case these little trees are just “stumps” and contain only a single decision.

The next parameter, the weight trim rate, is used to make the computation more efficient and therefore much faster. As training goes on, many data points become unimportant. That is, the weight D_t(i) for the ith data point becomes very small. The setWeightTrimRate() function sets a threshold, between 0 and 1 (inclusive), that is implicitly used to throw away some training samples in a given boosting iteration. For example, suppose weight trim rate is set to 0.95 (the default value). This means that the “heaviest” samples (i.e., samples that have the largest weights) with a total weight of at least 95% are accepted into the next iteration of training, while the remaining “lightest” samples with a total weight of at most 5% are temporarily excluded from the next iteration. Note the words “from the next iteration”—the samples are not discarded forever. When the next weak classifier is trained, the weights are computed for all samples and so some previously insignificant samples may be returned to the next training set. Typically, because of the trimming, only about 20% of samples or so take part in each individual round of training and therefore the training accelerates by factor of 5 or so. To turn this functionality off, call setWeightTrimRate(1.0).

For other parameters, note that cv::ml::BoostImpl inherits (via cv::ml::Boost) from cv::ml::DTrees, so we may set the parameters that are related to the decision trees themselves through the inherited interface functions. Overall, training of the boosting model and then running prediction is done in precisely the same way as with cv::ml::DTrees or essentially any other StatModel derived class from the ml module.

The code .../opencv/samples/cpp/letter_recog.cpp from the OpenCV package shows an example of the use of boosting. The training code snippet is shown in Example 21-2. This example uses the classifier to try to recognize a–z characters, starting with a public data set. That data set has 20,000 entries, each with 16 features and one “result.” The features are floating-point numbers and the result is a single character. Because boosting can only be used for two-class discrimination, this program uses the “unrolling” technique that we (briefly) encountered earlier. We will discuss that technique in more detail here.

In unrolling, the data set is essentially expanded from one set of training data to 26 sets, each of which is extended such that what was once the response is now added as a feature. At the same time, the new responses for these extended vectors are now just 1 or 0: true or false. In this way the classifier is being trained to, in effect, answer the question: is equal to by learning the relationships and ?²⁴ See Example 21-2.

...
cv::Mat var_type( 1, var_count + 2, CV_8U );    // var_count is # features (16 here)
var_type.setTo( cv::Scalar::all(VAR_ORDERED) );
var_type.at<uchar>(var_count) = var_type.at<uchar>(var_count+1) = VAR_CATEGORICAL;
...

The first thing to do is to create the array var_type that indicates how to treat each feature and the results (Example 21-2). Then the training data structure is created. Note that this is wider than you might expect by one. Not only are there var_count (in this case, this happens to be 16) features from the original data, there is the one extra column for the response, and there is one more column in between for the extension of the features to include what was once the alphabet-character response (before the unrolling).

cv::Ptr<cv::ml::TrainData> tdata = cv::ml::TrainData::create(
  new_data,                   // extended, 26x as many vectors, each contains y_i
  ROW_SAMPLE,                 // feature vectors are stored as rows
  new_responses,              // extended, 26x as many vectors, true or false
  cv::noArray(),              // active variable index, here just "all"
  cv::noArray(),              // active sample index, here just "all"
  cv::noArray(),              // sample weights, here just "all the same"
  var_type                    // extended, has 16+2 entries now
);

The next thing to do is to construct the classifier. Most of this is pretty usual stuff, but one thing that is unusual is the priors. Note that the price of getting a wrong answer has been inflated to 25 over the price of getting a wrong answer. This is not because some letters are poisonous, but because of the unrolling. What this is saying is that it is 25 times costlier to say that a letter is not something that it is, than to say that it is something that it is not. This needs to be done because there are 25× more vectors effectively enforcing the “negative” rules, so the “positive” rules need correspondingly more weight.²⁵

vector<double> priors(2);
priors[0] = 1;  // For false (0) answers
priors[1] = 25  // For true (1) answers

model = cv::ml::Boost::create();
model->setBoostType( cv::ml::Boost::GENTLE );
model->setWeakCount( 100 );
model->setWeightTrimRate( 0.95 );
model->setMaxDepth( 5 );
model->setUseSurrogates( false );

cout << "Training the classifier (may take a few minutes)...
";
model->setPriors( cv::Mat(priors) );

model->train( tdata );

The prediction function for boosting is also similar to that for decision trees, in this case using model->predict(). As described earlier, in the context of boosting, this method computes the weighted sum of weak classifier responses, takes the sign of the sum, and then converts it to the output class label. In some cases, it may be useful to get the actual sum value—for example, to evaluate how confident the decision is. In order to do that, pass the cv::ml::StatModel::RAW_OUTPUT flag to the predict method. In fact, that is what needs to be done in this case. When dealing with unrolled data, it is not so rare to get two (or more) “true” responses. In this case, one typically chooses the most confident answer.

Mat temp_sample( 1, var_count + 1, CV_32F ); // An extended sample "proposition"
float* tptr = temp_sample.ptr<float>();      // Pointer to start of proposition

double correct_train _answers = 0, correct_test _answers = 0;

for( i = 0; i < nsamples_all; i++ ) {

  int          best_class = 0;             // Strongest proposition found so far
  double       max_sum    = -DBL_MAX;      // Strength of current best prop
  const float* ptr        = data.ptr<float>(i);  // Points at current sample

  // Copy features from current sample into temp extended sample
  //
  for( k = 0; k < var_count; k++ ) tptr[k] = ptr[k];

  // Add class to sample proposition, then make a prediction for this proposition
  // If this proposition is more true than any previous one, then record this
  // one as the new "best".
  //
  for( j = 0; j < class_count; j++ ) {
    tptr[var_count] = (float)j;
    float s = model->predict(
      temp_sample, noArray(), StatModel::RAW_OUTPUT
    );
    if( max_sum < s ) { max_sum = s; best_class = j + 'A'; }
  }

  // If the strongest (truest) proposition matched the correct response, then
  // score 1, else 0.
  //
  double r = std::abs( best_class - responses.at<int>(i) ) < FLT_EPSILON ? 1 : 0;

  // If we are still in the train samples, record one more correct train result.
  // Otherwise, record one more correct test result.
  // Hope nobody shuffled the samples!
  //
  if( i < ntrain_samples )
    correct_train _answers += r;
  else
    correct_test  _answers += r;
}

Of course, this isn’t the fastest or most convenient method of dealing with many class problems. Random trees may be a preferable solution, and we will consider it next.

Random trees code

We are by now familiar with how the ML module works, and random trees are no exception. We start with the class declaration. Once again, there is a class cv::ml::RTreesImpl that inherits from cv::ml::RTrees, which itself inherits from the cv::ml::DTrees class:

// Somewhere above..
// namespace cv {
//   namespace ml {
//
class RTreesImpl: public RTrees {  // cv::ml::RTrees is derived from cv::ml::DTrees

public:

  // whether variable importance should be computed during the training
  // makes the training noticeably slower
  //
  bool getCalculateVarImportance() const;          // Get if importance should be
                                                   //  computed during training
  int  getActiveVarCount() const;                  // Get N Var for each split
  TermCriteria getTermCriteria() const;            // Get max N trees or accuracy

  void setCalculateVarImportance( bool val );      // Get if importance should be
                                                   //  computed during training
  void setActiveVarCount( int val );               // Set N Var for each split
  void setTermCriteria( const TermCriteria& val ); // Set max N trees or accuracy

  ...

  Mat getVarImportance() const;

  ...

  // from class RTrees
  //
  //static Ptr<RTrees> create(); // The algorithm "constructor",
  //                             //  returns Ptr<RTreesImpl>
};

A new method, setCalculateVarImportance(), is a switch to enable calculation of the variable importance of each feature during training (at a slight cost in additional computation time).

Figure 21-8 shows the variable importance computed on a subset of the mushroom data set that we discussed earlier agaricus-lepiota.data.

Figure 21-8. Variable importance over the mushroom data set for random trees, boosting, and decision trees: random trees used fewer significant variables and achieved the best prediction (100% correct on a randomly selected test set covering 20% of data). Note that in OpenCV 3.x one can explicitly get the variable importance only for RTrees, and not for decision trees or boosting

The setActiveVarCount() method sets the size of the randomly selected subset of features to be tested at any given node and, if not set by the user explicitly, is typically set to the square root of the total number of features.

Using setTermCriteria(), you can set the termination criteria that contains both the maximum number of trees (cv::TermCriteria::maxIter) and the OOB error, below which the tree generating need not continue (cv::TermCriteria::epsilon). As usual, either or both of the usual two stopping criteria can be applied (usually it’s both: cv::TERMCRIT_ITER | cv::TERMCRIT_EPS). Otherwise, random trees training has the same form as decision trees training, boosting training, and so on.

The same multiclass learning example that we looked at before in the case of boosting, .../opencv/samples/cpp/letter_recog.cpp, also provides for training random forests. The following excerpt shows some salient points. Note that, unlike with boosting, we can train directly on the multiclass data.

using namespace cv;
...
Ptr<ml::RTrees> forest = ml::RTrees::create();
forest->setMaxDepth(10);
forest->setMinSampleCount(10);
forest->setMaxCategories(15);
forest->setCalculateVarImportance(true);
forest->setActiveVarCount(4);
forest->setTermCriteria(
  TermCriteria(
    TermCriteria::MAX_ITER+TermCriteria::EPSILON,
    100,
    0.01
  )
);
forest->train(tdata, 0);

...

Random trees prediction is no different from all other models from ml. Here’s an example prediction call from the letter_recog.cpp file:

...

for( int i = 0; i < nsamples_all; i++ ) {

  cv::Mat sample = mydata.row( i );

  float r = forest->predict( &sample );
r = fabs( (float)r - responses.at<float>[i]) <= FLT_EPSILON ? 1 : 0;

// Accumulate some statistics using 'r'
  if( i < ntrain_samples )
    correct_train _answers += r;
  else
    correct_test  _answers += r;

}

In this code, the return variable r is converted into a count of correct predictions. You can compute the same statistics using the universal method cv::ml::StatModel::calcError().

Finally, there are random tree analysis and utility functions. Assuming that setCalculateVarImportance( true ) was called before training, we can obtain the relative importance of each variable using the cv::ml::RTrees member function:

cv::Mat RTrees::getVarImportance() const;

In this case, the return will be a vector containing the relative importances of each of the features.

Using random trees

We’ve remarked that the random trees algorithm often performs the best (or among the best) on the data sets we tested, but the best policy is still to try many classifiers once you have your training data defined. We ran random trees, boosting, and decision trees on the mushroom data set. From the 8,124 data points we randomly extracted 1,624 test points, leaving the remainder as the training set. After training these three tree-based classifiers with their default parameters, we obtained the results shown in Table 21-3 on the test set. The mushroom data set is fairly easy and so—although random trees did the best—it wasn’t such an overwhelming favorite that we can definitively say which of the three classifiers works better on this particular data set.

What is more interesting is the variable importance (which we also measured from the classifiers), shown in Figure 21-8. The figure shows that random trees and boosting each used significantly fewer important variables than required by decision trees. Above 15% significance, random trees used only 3 variables and boosting used 6, whereas decision trees needed 13. We could thus shrink the feature set size to save computation and memory and still obtain good results. Of course, for the decision trees algorithm you have just a single tree, while for random trees and AdaBoost you must evaluate multiple trees; thus, which method has the least computational cost depends on the nature of the data being used.

Expectation maximization with cv::EM()

In OpenCV, the EM algorithm is implemented in the cv::ml::EM class, which has the following declaration:

// Somewhere above..
// namespace cv {
//   namespace ml {
//
class EMImpl: public EM {       // cv::ml::EM is derived from cv::ml::StatModel

public:

  // Types of covariation matrices
  // (Inherited from cv::ml::EM)
  //
  //enum Types {
  //  COV_MAT_SPHERICAL = 0,
  //  COV_MAT_DIAGONAL  = 1,
  //  COV_MAT_GENERIC   = 2,
  //  COV_MAT_DEFAULT   = COV_MAT_DIAGONAL
  //};

  // get/set the number of clusters/mixtures (5 by default)
  //
  int  getClustersNumber() const;                  // Get number of clusters
  int  getCovarianceMatrixType() const;            // Get cov. mtx. type
  TermCriteria getTermCriteria() const;            // Get max iter/accuracy

  void setClustersNumber( int val ) ;              // Set number of clusters
  void setCovarianceMatrixType( int val );         // Set cov. mtx. type
  void setTermCriteria( const TermCriteria& val ); // Set max iter/accuracy

  ...

  // the prediction method, see below
  //
  Vec2d predict2(
    InputArray  sample,
    OutputArray probs
  ) const;
  bool trainEM(...); // see below
  bool trainE(...);  // see below
  bool trainM(...);  // see below

  ...

  // from class EM
  //
  //static Ptr<EM> create(); // The algorithm "constructor",
  //                         //  returns Ptr<EMImpl>

};

When constructing and configuring a cv::EM object, we must tell the algorithm how many clusters there will be. We do this with the setClusterNumber() method. As in the case of the K-means algorithm, it is always possible to do tests with different numbers of clusters separately, but the basic cv::EM object can handle only one specific number of clusters at a time.

The next member function, setCovarianceMatrixType(), specifies the constraints that you wish to have the EM algorithm apply to the covariance matrices associated with the individual components of the Gaussian mixture model. This argument must take one of three possible values:

• cv::ml::EM::COV_MAT_SPHERICAL

• cv::ml::EM::COV_MAT_DIAGONAL

• cv::ml::EM::COV_MAT_GENERIC

In the first case, each mixture component is assumed to be rotationally symmetric. This means that it has only one free parameter that can be maximized in the M-step. Each covariance is just that parameter multiplied by an identity matrix. In the second case, cv::EM::COV_MAT_DIAGONAL (which is the default), each matrix is expected to be diagonal, and so the number of parameters is equal to the number of dimensions of the matrix (i.e., the dimensionality of the data). Finally, there is the case of cv::EM::COV_MAT_GENERIC, where each covariance matrix is characterized by the variables needed to characterize an arbitrary symmetric matrix. In general, the complexity of the model that can be trained is strongly dependent on the amount of data available. In practice, if one has very little data, cv::EM::COV_MAT_SPHERICAL is probably a good idea. Conversely, unless one has a vast amount of data, cv::EM::COV_MAT_GENERIC is probably not a good idea.

The final configuration that EM requires is the termination criteria, set by means of setTermCriteria(). The termCrit argument required is of the usual cv::TermCriteria type, and specifies the maximum number of iterations allowed and (or) the maximum change in likelihood that will be considered “small enough” to terminate the algorithm.

Once you have created an instance of the cv::EM object and set the parameters, you can train it with one of three train*() methods specific to the EM algorithm—trainEM(), trainE(), and trainM().

bool cv::ml::EM::train(
  cv::InputArray  samples,
  cv::OutputArray logLikelihoods = cv::noArray(),
  cv::OutputArray labels         = cv::noArray(),
  cv::OutputArray probs          = cv::noArray()
);

virtual bool cv::ml::EM::trainE(
  cv::InputArray  samples,
  cv::InputArray  means0,
  cv::InputArray  covs0          = cv::noArray(),
  cv::InputArray  weights0       = cv::noArray(),
  cv::OutputArray logLikelihoods = cv::noArray(),
  cv::OutputArray labels         = cv::noArray(),
cv::OutputArray probs            = cv::noArray()
);

bool cv::ml::EM::trainM(
  cv::InputArray  samples,
  cv::InputArray  probs0,
  cv::OutputArray logLikelihoods = cv::noArray(),
  cv::OutputArray labels         = cv::noArray(),
cv::OutputArray probs            = cv::noArray()
);

The trainEM() method of cv::EM expects the usual input array of samples and returns the responses (labels), the likelihoods (logLikelihoods), and the assignment probabilities (probs). The responses appear in the labels array, which will have a single column containing one row for each data point. The value on the ith row will be an integer identifying the cluster to which that data point was assigned. This integer is the largest of the membership probabilities computed across the clusters (set by nclusters when you called the constructor). The array probs will actually contain these individual probabilities, with each row giving the probabilities for a given point, and the column being the probability for that cluster (i.e., probs.at<int>(i,k) is the probability of point i being a member of cluster k). Finally, the likelihoods associated with each point are returned in the logLikelihoods array; in essence these likelihoods indicate (in a relative sense) how probable an individual observation was under the final model. As the name suggests, the values returned are actually the natural logarithm of the likelihoods. Any of these three outputs may be replaced with cv::noArray() if they are not needed. Once the model is trained, it can be used for prediction.

In addition to the train method just described, there are two additional methods, trainE() and trainM(). These methods start the algorithm off in the E- or M-step (respectively) and provide, in the case of trainE(), an initial model to use, or in the case of trainM(), a set of initial cluster assignment probabilities.

In the case of trainE(), the model is supplied in the form of the input arrays means0, covs0, and weights0. The form of the means should be an array with rows and columns, where is the number of clusters and is the dimensionality of the sample data. The covariances should be supplied as an STL vector containing separate arrays, each being × and containing the covariance matrix for Gaussian component k. Finally, the array weights0 should have a single column and rows. The entry in the kth row should be the mixture probability associated with Gaussian component k (i.e., the marginal probability that any random sample will be drawn from component k, as opposed to some other component).

In the case of trainM(), you must supply probs0, the membership probabilities associated with each point, in the same form that they would be computed by train() as just described. This means that probs0 is an × array with one row for each sample and the membership probabilities for each Gaussian component in the corresponding columns for that data point.

Once you have trained your model, you can use the cv::EM::predict() method to have the trained algorithm attempt to predict what cluster a novel point would most likely be a member of. The return value of predict() will be of type cv::Vec2d. The first component of the returned vector will give the probability associated with the assignment under the current model, while the second will give the cluster label.

Using K-nearest neighbors with cv::ml::KNearest()

The KNN algorithm is implemented in OpenCV by the cv::ml::KNearest class. The class declaration for cv::ml::KNearest has the following form:

// Somewhere above..
// namespace cv {
//   namespace ml {
//
class KNearestImpl: public KNearest { // cv::ml::KNearest is derived
                                      // from cv::ml::StatModel
public:

  // (Inherited from cv::ml:: KNearest)
  //
  //enum Types {
  //  BRUTE_FORCE = 1,
  //  KDTREE      = 2
  //};

  // use K-nearest for regression or for classification

  //
  int  getDefaultK() const;         // Get the default number of neighbors
  bool getIsClassifier() const;     // Get if classifier, else it is regression
  int  getEmax() const;             // Get max "close" neighbors (KDTree)
  int  getAlgorithmType() const;    // Get whether brute-force or KD-Tree search
                                    //  (will be either BRUTE_FORCE or KDTREE)

  void setDefaultK( int val );      // Set the default number of neighbors
  void setIsClassifier( bool val ); // Set if classifier, else it is regression
  void setEmax( int val );          // Set max "close" neighbors (KDTree)
  void setAlgorithmType( int val ); // Set whether brute-force or KD-Tree search
                                    //  (must be either BRUTE_FORCE or KDTREE)
  ...

  // find the k nearest neighbors for each sample
  //
  float findNearest(
    InputArray  samples,
    int         k,
    OutputArray results,
    OutputArray neighborResponses = noArray(),
    OutputArray dist              = noArray()
  ) const = 0;

  ...

  // from class KNearest
  //
  //static Ptr<KNearest> create(); // The algorithm "constructor",
  //                               //  returns Ptr<KNearestImpl>
};

The method setDefaultK() sets the number of neighbors to be considered if you plan to use cv::ml::StatModel::predict instead of cv::ml::KNearest::findNearest(), where this parameter is specified explicitly (k corresponds to in our preceding discussion of the KNN algorithm).

The setIsClassifier() function is used if you are using KNN for regression (i.e., you plan to approximate some function using the discrete training set). In this case, you should call setIsClassifier(false) prior to calling the train() method.

You train the KNN model as usual by calling the cv::ml::StatModel::train() method. The trainData input is the usual array containing rows and columns, with being the number of samples and being the number of dimensions of the data. The responses array must contain the usual rows and a single column. How the algorithm then handles this data depends on the setAlgorithmType() method. You can call this method with either BRUTE_FORCE or KDTREE.

If the algorithm is set to BRUTE_FORCE, this training data is simply stored internally as an array and then scanned sequentially in order to find the nearest neighbors. In the case of KDTree, the BBF (best-bin-first) algorithm (introduced by D. Lowe) will be used, which is much more efficient when .

It’s important to note that unlike many other models in the ml module, the KNN model can be updated with new trained data after it’s trained. In order to do this, pass the flag cv::ml::StatModel::UPDATE_MODEL to the train() method.

Once the data is trained, you can use your cv::ml::KNearest object to make predictions about it. The cv::ml::KNearest object provides the usual predict() method as well as the more model-specific findNearest(), which is equivalent to findNearest( samples, getDefaultK(), results, noArray(), noArray() ). But the method findNearest() method allows you to retrieve some additional information about the neighbors. The findNearest() method has the following definition:

virtual float cv::ml::KNearest::findNearest(
  cv::InputArray  samples,
  int             k,
  cv::OutputArray results,
  cv::OutputArray neighborResponses = cv::noArray(),
  cv::OutputArray dist              = cv::noArray()
) const = 0;

The method takes one or more samples at a time, with the samples argument being any number of rows with each row containing a single input. The value of k that is used for the comparison (the argument k) can be any value up to the value of given when the object was trained. The predictions will be placed in the array results, which will have a single column and one row corresponding to each point (row) in samples. If provided, the arrays neighborResponses and dists will be filled with the responses from and distances to the various neighbors identified for each query point. Each of these will have one row per point (row) in samples and one column for each of the k neighbors found for that point.³⁰

Both methods will return a single floating-point value; this is used when the number of query points in samples is just one so only one computed response will be returned. In this case, you do not need to provide a results array at all.

Back propagation

Refer to Figure 21-11. We can use a trained back-propagation network to make decisions or to fit functions by feeding an input in at the “bottom” that is weighted and transformed through nonlinearities (usually sigmoid or rectifier functions). This signal is propagated up layer by layer in the network until an output is produced at the “top.” In training mode, we use a loss function to gauge how well the network is doing. In Figure 21-11, this loss function is the square of target value minus the actual value. The essential issue that back propagation addresses is how to update the weights in the neural network so that the loss function is minimized over the data set.

To minimize the loss function, the chain rule in calculus is used to derive, from top-to-bottom (“back propagation”), how the weights must change in relation to the loss function at the top.³⁵ The back-propagation algorithm is a message-passing algorithm that takes advantage of the topology of the multilayer perceptron. It first computes the error at the output nodes (the difference between what you want and what you get). It then propagates this information backward to those nodes that feed the output nodes. They combine the information they received from the output nodes to compute the necessary derivatives relative to their own weights. This information is then passed on again to the next layer toward the input layer. In this way, we can compute the portion of the gradient that pertains to each weight locally, using only information about that node and information passed to that node from the layer one step closer to the output to which it is connected.

Figure 21-11. Back-propagation setup. The goal is to update the weights in order to minimize the loss function. Each hidden and output node gets weighted activation , which then goes through a nonlinearity function g or f. In practice, these nonlinear functions are all the same, often a sigmoid or a rectifier function, but for generality, they can be different

In the network of Figure 21-11, the output weight change function is:

The hidden unit weight change function is:

where represents the target values, and the input, hidden, and outlook activations are . The last two activations are transformed by the nonlinear functions g and f having input weights , respectively. This forms the basic core of the back-propagation algorithm.

The Rprop algorithm

The Rprop (resilient back propagation) was introduced by Martin Riedmiller and Heinrich Braun [Riedmiller93] and provides an often-superior method of doing the update step of the back-propagation algorithm just described. The essential difference between back propagation and Rprop is that, while back propagation explicitly computes the magnitude of each step by which each weight is updated, Rprop uses only the sign (direction) of the computed update. When computing the magnitude of the update, Rprop computes and stores the direction of the update, then makes a change by a standard step (called ). On the next pass, if the direction changes (meaning that the algorithm essentially overshot the goal), the step is reduced by a constant scale factor (usually called ). If the step does not change sign, then the algorithm increases the step size of the next iteration by multiplying by a different scale factor (usually called ). Clearly must be less than 1 and must be greater than 1; canonical values for these parameters are 0.5 and 1.2, respectively. In the OpenCV implementation, the step is never allowed to be smaller than some absolute minimum, nor larger than some absolute maximum (called and , respectively).

Using artificial neural networks and back propagation with cv::ml::ANN_MLP

The OpenCV implementation of multilayer perceptron learning handles the construction and evaluation of the network, and implements the back-propagation algorithm (or Rprop algorithm) just described for training the network. All of this is contained within the cv::ml::ANN_MLP class, which implements the usual interface inherited from the statistical learning base class. The cv::ml::ANN_MLP class has the usual hidden implementation class, in this case with the following definition:

// Somewhere above..
// namespace cv {
//   namespace ml {
//
class ANN_MLPImpl: public ANN_MLP {   // cv::ml::ANN_MLP is derived
                                      // from cv::ml::StatModel
public:

  // (Inherited from cv::ml::ANN_MLP)
  //
  //enum TrainingMethods {
  //  BACKPROP        = 0, // The back-propagation algorithm.
  //  RPROP           = 1  // The RPROP algorithm.
  //};
  //
  //enum ActivationFunctions {
  //  IDENTITY        = 0,
  //  SIGMOID_SYM     = 1,
  //  GAUSSIAN        = 2
  //};
  //
  //enum TrainFlags {
  //  UPDATE_WEIGHTS  = 1,
  //  NO_INPUT_SCALE  = 2,
  //  NO_OUTPUT_SCALE = 4
  //};


  int getTrainMethod() const;         // Get backpropagation or RPROP method
  int getActivationFunction() const;  // Get the activation function
                                      // (IDENTITY, SIGMOID_SYM or GAUSSIAN)
  Mat getLayerSizes() const;               // Get size of all the layers
  TermCriteria getTermCriteria() const;    // Get max it / reproject error
  double getBackpropWeightScale() const;   // Set Backprop parameter
  double getBackpropMomentumScale() const; //  "
  double getRpropDW0() const;              // Set Rprop parameter
  double getRpropDWPlus() const;           //  "
  double getRpropDWMinus() const;          //  "
  double getRpropDWMin() const;            //  "
  double getRpropDWMax() const;            //  "

  void setTrainMethod(                // Set backpropagation or RPROP method
    int    method,                    // either of: BACKPROP or RPROP
    double param1 = 0,
    double param2 = 0
  );
  void setActivationFunction(         // Set activation function
    int    type,                      // IDENTITY, SIGMOID_SYM, or GAUSSIAN
    double param1 = 0,
    double param2 = 0
  );
  void setLayerSizes( InputArray _layer_sizes ); // Set size of all the layers
  void setTermCriteria( TermCriteria val );      // Set max it / reproject error
  void setBackpropWeightScale( double val );     // Get Backprop parameter
  void setBackpropMomentumScale( double val );   //  "
  void setRpropDW0( double val );                // Set Rprop parameter
  void setRpropDWPlus( double val );             //  "
  void setRpropDWMinus( double val );            //  "
  void setRpropDWMin( double val );              //  "
  void setRpropDWMax( double val );              //  "

  ...

  Mat getWeights( int layerIdx ) const; // Get the computed weights for
                                        //  the interlayer connections
  ...

  // from class ANN_MLP
  //
  //static Ptr<ANN_MLP> create(); // The algorithm "constructor",
  //                              //  returns Ptr<ANN_MLPImpl>
};

To use the multilayer perceptron, you first construct the empty network with the cv::ml::ANN_MLP::create() method, and then set the size of all the layers. Next, once you have set all the parameters (activation function, the training method, and its parameters), you can finally call the cv::ml::StatModel::train() method, as usual.

The argument of the setLayerSizes() method specifies the basic structure of the network.³⁶ This should be a single-column array in which the rows specify the number of inputs, the number of nodes in the first layer, the number of nodes in the second layer, and so on, until finally, the last row gives the number of nodes in the final layer (and thus the number of outputs from the network).³⁷

The method setActivationFunction() specifies the function that is applied to the weighted sum of the inputs. The computed output of any node in the network is given by some function of the weighted sum of inputs plus an offset term: . The weights and the offset term will be learned in the training phase, but the function itself is a property of the network. At this time, there are three functions you can choose from: a linear function, a sigmoid, and a Gaussian function.³⁸ The values and corresponding arguments for setActivationFunction are shown in Table 21-4.

Table 21-4. Options for the artificial neuron activation function used by cv::ANN_MLP. Each is a function of

Value of activateFunc	Activation function
cv::ANN_MLP::IDENTITY
cv::ANN_MLP::SIGMOID_SYM
cv::ANN_MLP::GAUSSIAN

The final two arguments, fparam1 and fparam2, correspond to the variables and in Table 21-4. Unless you are an expert however, in almost all cases, you will want to use the default sigmoid activation function, and in the majority of those, you will want to set these parameters at their generic values (of 1.0).

Once you have created your artificial neural network, you will want to train the network with the data you have. In order to train the network, you will need to construct cv::ml::TrainData as usual, and then pass it to the cv::ml::StatModel::train() method, overridden in cv::ml::ANN_MLP. Unlike all other models from ml, neural nets are able to handle vector outputs, not only scalar outputs, so that the response is not necessarily a single-column vector—it can be a matrix, one row per sample. And the vector output can actually be used to implement a multiclass classifier using neural networks.

Also, neural networks can handle the nonempty sampleWeights vector, passed to cv::ml::TrainData::create, which allows you to assign a relative importance to each data sample (row) in inputs and outputs. Each row in sampleWeights can be set to an arbitrary positive floating-point value, corresponding to the data in the same rows of inputs and outputs. The weights are automatically normalized, so all that matters is their relative sizes. The sampleWeights argument is only respected by the Rprop algorithm, so if you are using back propagation it will be ignored (see next discussion).

Parameters for training

The method setTermCriteria() specifies when training will terminate, and has the same meaning if you are using back propagation or RProp (see setTrainMethod(), next). The number of iterations in the termination criteria is exactly the maximum number of steps the update will take. The epsilon portion of the termination criteria sets the minimum change in the reprojection error required for the iteration to continue; this reprojection error is equal to 0.5 times the sum of the squared differences between training set results and computed outputs over the entire training set.³⁹

You can set train_method to either cv::ANN_MLP::BACKPROP or cv::ANN_​MLP::RPROP. By default, this parameter will be set to RPROP, as this method is generally more effective for training the network in most circumstances. In the case of RPROP, param1 and param2 in the constructor correspond to the initial update step size and the smallest update step size (respectively). In the case of BACKPROP, param1 and param2 correspond to what are called the weight gradient and the momentum, respectively. Alternatively, you may set these parameters using the setBackpropWeightScale() and setBackpropMomentumScale() member accessor functions. The weight gradient value multiplies the weight gradient term in the back-propagation update step and essentially controls how fast the updates move in the desired direction. A typical value for this term is 0.1. The momentum value multiplies an additional term in the update that is proportional to the difference in the value between the prior step and the step prior to that—giving something like a velocity for the weight. The momentum term premultiplies this velocity, which has the effect of smoothing out large fluctuations in the update. If set to 0, this term is effectively eliminated. However, a small value (typically about 0.1 also) often gives a substantially faster overall convergence. In addition to these two parameters, there are the accessors setRpropDWPlus(), setRpropDWMinus(), setRpropDW0(), setRpropDWMin(), and setRpropDWMax(), which can be used to set the values , , , , and respectively (as well as the corresponding get*() methods to simply inspect these values).

Once you have trained your network, you can then use it to make predictions in the usual way using the overridden cv::ml::StatModel::predict() method. As usual, samples must be an array with one row per input data point and the correct number of columns (the same number as the training data). The results array will have as many rows as inputs did, but the number of columns equal to the number of nodes in the output layer of the network. The return value is meaningless and can be ignored.⁴⁰

About kernels

The kernel is the mapping that takes points from the native space of the input training vectors to the higher-dimensional feature space and back again. Remember that it is in this higher-dimensional space that we intend to find the separating hyperplane. The kernel is composed of two parts: the part that takes us from the space of the input features to the kernel space and the inverse transformation that takes us back. Customarily, the kernel is written as , while the mappings are written as and , respectively. The kernel, by definition, can then be expressed as:

It is easiest to understand a kernel by way of example. One of the kernels available to OpenCV’s SVM is called the polynomial kernel. This kernel has the form:

for some value of c > 0 and integer q > 0. The mapping that corresponds to this kernel can have a very large number of components. Even for the simplest case where q = 2 and c = 0, given an input vector of three dimensions (), with :⁴³

For comparison, for c = :

Notice that for the first choice of kernel, corresponds to a kernel space of dimension of ; for the second kernel with the same input dimension, .

What is particularly important about these kernels, however, is that the mappings never need to be computed in the course of computing the separation hyperplane for the SVM. The reason for this critical fact is that all of the computations in the feature space that are needed only involve dot products between the points. This means that, wherever we find , it will be in the form for some pair of points and . Worded differently, we never need to evaluate ; we only need to evaluate . The implications of this are profound, since even though the dimension of the kernel space might be very high (even infinite!), we need only be able to evaluate for vectors of the input feature space dimension to train and use the SVM. This property is called the kernel trick in SVM literature.

In practice then, one only needs to select a kernel and from there, the SVM can do the work. The actual kernel selected will have a heavy influence on how the constructed hyperplane is generated, how well it separates the training data, and how well that separation generalizes to future data. OpenCV currently provides six different kernels you can use in your support vector machines (see Table 21-5).

Table 21-5. The available kernel functions for the OpenCV SVM implementation

Kernel	OpenCV name (kernel_type)	Kernel function	Parameters
Linear	cv::ml::SVM::LINEAR
Polynomial	cv::ml::SVM::POLY		degree (q), gamma (γ > 0), coef0 (c0)
Radial basis functions	cv::ml::SVM::RBF		gamma (γ > 0)
Sigmoid	cv::ml::SVM::SIGMOID		gamma (γ > 0), coef0 (c0)
Exponential chi-squared	cv::ml::SVM::CHI2		gamma (γ > 0)
Histogram intersection	cv::ml::SVM::INTER

Handling outliers

OpenCV supports two different variants of SVM called C-vector SVM and v-vector SVM.⁴⁴ These extensions support the possibility of outliers. In this context, an outlier is a data point that cannot be assigned to the correct class (or equivalently a data set that cannot be separated by a hyperplane in the kernel space). In addition to outliers, the implementation of both of these methods in OpenCV allows for multiclass () classification.

The C-vector SVM, also known as a soft margin SVM, allows for the possibility that any particular point is an outlier by penalizing the outlier by an amount proportional to the distance that point is found to be past the decision boundary. Such distances are customarily called slack variables, while this constant of proportionality is conventionally called C (the latter being the origin of the name for this method).

The second extension, which also allows for the possibility of outliers, is called the v-vector SVM [Schölkopf00]. In this case, the proportionality constant analogous to C takes a constant (predetermined) value. Instead, however, there is the new parameter v. This parameter sets an upper bound on the number of training errors (i.e., misclassifications), as well as a lower bound on the fraction of training points that are support vectors. The parameter v must be between 0.0 and 1.0.

Comparing the two, the C-vector SVM has a more straightforward implementation, and so is often faster to train. However, because there is no natural interpretation of C, it is difficult to find a good value other than through trial and error. On the other hand, the parameter of the v-vector SVM has a natural interpretation, and so in practice is often easier to use.

Multiclass extension of SVM

Both the C-vector SVM and the v-vector SVM, as implemented in OpenCV, support the possibility of more than two classes. Though there are many possible ways to do this, OpenCV uses the method called one-versus-one.⁴⁵ In this method, if there are k classes, a total of k(k – 1)/2 classifiers are trained. When an object is to be classified, every classifier is run and the class that has the majority of wins is taken to be the result.

One-class SVM

Another interesting variant of the SVM is the one-class SVM. In this case, all of the training data is taken to be exemplary of a single class, and a decision boundary is sought that separates that one class from everything else. As with the multiclass extensions just described, the one-class SVM also supports outliers with a mechanism of slack variables similar to the C-vector SVM (and having the same parameter).

Support vector regression

In addition to extensions for single class and multiclass, support vector machines have also been extended to allow for the possibility of regression (as opposed to classification). In the regression case, the input results are sequential values, rather than class labels, and the output is an interpolation (or extrapolation) based on the input values which were provided. OpenCV supports two different algorithms for support vector regression (SVR): -SVR [Drucker97] and v-SVR [Schölkopf00].

In its most abstract form, the SVR and the SVM are essentially the same, with the exception that the error function that is being minimized in SVM always has the output equal to one of two values (typically +1 and –1), while the SVR has a different objective for each input example. The nuance, however, arises from how the slack variables are handled, and what points are considered “outliers.” Recall that in the SVM, points were assigned to classes that were separated by the maximum margin hyperplane. Thus, being an outlier was a statement about the location of a point, given its label, relative to the hyperplane. In the SVR case, the hyperplane is the model for the function. (Recall that this plane in the kernel space can be very complicated in the input feature space.) Thus, it is the points that are more than some distance from the hyperplane that are outliers (in either direction). It is how this distance is handled that differentiates the two algorithms we have available to us for SVR in OpenCV.

The first form algorithm, -SVR, uses a parameter called that defines a space around the hyperplane within which no cost is assigned to predictions being off by this much or less. Beyond this distance, a cost is assigned that grows linearly with distance. This is true for points both above and below the hyperplane.

The second algorithm, v-SVR, has a relationship with -SVR similar to that between v-SVM and C-SVM. Specifically, the v-SVR uses the parameter v to set the minimum fraction of input vectors that will be support vectors. In this manner, the meaning of the parameter v in the v-SVR, as with the v-SVM, is substantially more intuitive, and therefore easier to set to a sensible value.

Using support vector machines and cv::ml::SVM()

The SVM classification interface is defined in OpenCV by the cv::ml::SVM class in the ML library. cv::ml::SVM, like the other objects we have looked at in this section, is derived from cv::ml::StatModel class, and itself serves as the interface to an implementation class, in this case called cv::ml::SVMImpl:

// Somewhere above..
// namespace cv {
//   namespace ml {
//
class SVMImpl: public SVM {   // cv::ml::SVM is derived
                              // from cv::ml::StatModel
public:

  // (Inherited from cv::ml::SVM)
  //
  //enum Types {
  //  C_SVC     = 100,
  //  NU_SVC    = 101,
  //  ONE_CLASS = 102,
  //  EPS_SVR   = 103,
  //  NU_SVR    = 104
  //};
  //
  //enum KernelTypes {
  //  CUSTOM    = -1,
  //  LINEAR    = 0,
  //  POLY      = 1,
  //  RBF       = 2,
  //  SIGMOID   = 3,
  //  CHI2      = 4,
  //  INTER     = 5
  //};
  //
  //enum ParamTypes {
  //  C         = 0,
  //  GAMMA     = 1,
  //  P         = 2,
  //  NU        = 3,
  //  COEF      = 4,
  //  DEGREE    = 5
  //};

  class Kernel : public Algorithm {

  public:

    int getType() const;
    void calc(
      int          vcount,
      int          n,
      const float* vecs,
      const float* another,
      float*       results
    );
  };

  int getType() const;                        // get the SVM type (C-SVM, etc.)
  double getGamma() const;                    // get the gamma param of kernel
  double getCoef0() const;                    // get the coeff0 param of kernel
  double getDegree() const;                   // get degree param of kernel
  double getC() const;                        // get C param of C-SVM or eps-SVR
  double getNu() const;                       // get nu param of nu-SVM or nu-SVR
  double getP() const;                        // get P param of eps-SVR
  cv::Mat getClassWeights() const;            // get the class priors
  cv::TermCriteria getTermCriteria() const;   // get training term criteria
  int getKernelType() const;                  // get the kernel type

  void setType( int val );                    // set the SVM type (C-SVM, etc.)
  void setGamma( double val );                // set the gamma param of kernel
  void setCoef0( double val );                // set the coeff0 param of kernel
  void setDegree( double val );               // set degree param of kernel
  void setC( double val );                    // set C param of C-SVM or eps-SVR
  void setNu( double val );                   // set nu param of nu-SVM or nu-SVR
  void setP( double val );                    // set P param of eps-SVR
  void setClassWeights( const Mat &val );     // set the class priors
  void setTermCriteria( const TermCriteria &val ); // set training term criteria
  void setKernel( int kernelType );           // set the kernel type

  ...

  // set the custom SVM kernel if needed
  //
  void setCustomKernel( const Ptr<Kernel> &_kernel );

  Mat      getSupportVectors() const;
  Mat      getUncompressedSupportVectors() const;
  double   getDecisionFunction(
    int         i,
    OutputArray alpha,
    OutputArray svidx
  );

  ParamGrid getDefaultGrid( int param_id ); // return default grid for any param
                                            // Choose from SVM::ParamTypes above
  bool trainAuto(
    const Ptr<TrainData>& data,
    int                   kFold      = 10,
    ParamGrid             Cgrid      = getDefaultGrid( C ),
    ParamGrid             gammaGrid  = getDefaultGrid( GAMMA ),
    ParamGrid             pGrid      = getDefaultGrid( P ),
    ParamGrid             nuGrid     = getDefaultGrid( NU ),
    ParamGrid             coeffGrid  = getDefaultGrid( COEF ),
    ParamGrid             degreeGrid = getDefaultGrid( DEGREE ),
    bool                  balanced   = false
  );

  ...

  // from class SVM
  //
  //static Ptr<SVM> create(); // The algorithm "constructor",
  //                          //  returns Ptr<SVMImpl>
};

As usual, you first create the instance of the class using the static create() method, then configure its parameters using the “setters,” and then run the train method. The first method, setType(), determines the SVM or SVR algorithm that is to be used. It can be any of the five values shown in Table 21-6. The next important method is setKernelType(), which may be any of the values from Table 21-5.

Table 21-6. The available types for the SVM and the corresponding values for the setType() method of cv::ml::SVM. The rightmost column lists the related properties of cv::ml::SVM that could be set for the SVM type along with the values to which they correspond (as described in the previous section)

SVM type	OpenCV name (svm_type)	Parameters
C-SVM classifier	cv::ml::SVM::C_SVC	C (C)
v-SVM classifier	cv::ml::SVM::NU_SVC	Nu (v)
One-class SVM	cv::ml::SVM::ONE_CLASS	C (C),	Nu (v)
-SVR	cv::ml::SVM::EPS_SVR	P (),	C (C)
v-SVR	cv::ml::SVM::NU_SVR	Nu (v),	C (C)

Once you have selected the type of SVM and the kernel type you would like to use, you should configure the kernel parameters with their associated set*() methods. The Parameters columns of Tables 21-5 and 21-6 indicate which parameters are needed in which cases, and which parameters in the equations of the previous sections are associated with each parameter. The default values of degree, gamma, coef0, C, Nu, and P are 0.0, 1.0, 0.0, 1.0, 0.0, and 0.0, respectively.

The setClassWeights() method allows you to supply a single-column array that provides an additional weighting factor for the slack variables. This is used only by the C-SVM classifier. The values you supply will multiply C for each individual training vector. In this way, you can assign a greater importance to some subset of the training examples, which will in turn help to guarantee that if any training vectors cannot be classified correctly, it will not be those. By default, class weights are not used.

The final method is setTermCriteria(), which can almost always be safely left at its default of 1000 iterations and FLT_EPSILON.

At this point, you might be looking at all of the parameters to the SVM and the kernels and wondering how you could possibly choose the right values for all of these things. If so, you would not be alone. In fact, it is common practice to simply step through ranges of all of these parameters and find the one that works the very best on the available data. This process is automated for you by the cv::ml::SVM::trainAuto() method.

When you are using cv::ml::SVM::trainAutio(), the train data you supply is the same as you would supply to cv::ml::StatModel::train(). The kFold argument controls how the validation is done for each parameter set. The method used is k-fold cross-validation (so the parameter should probably just be called k), by which the data set is automatically divided into k-subsets and then run k times; each time it is run, a different subset is held back for validating after training on the other (k – 1) subsets.

The next six parameters control the grids, which are the sets of values that will be tested for each of the parameters. A grid is a simple object with three data members: minVal, maxVal, and logStep. If you set the step to something less than 1, then the grid will not be used and instead the relevant value from params will be used for all runs. Typically, you will not grid-search over every parameter. You can construct a grid with the constructors:

cv::ml::ParamGrid::ParamGrid() {
  minVal  = maxVal = 0;
  logStep = 1;
}
cv:: ml::ParamGrid::ParamGrid(
  double minVal,
  double maxVal,
  double logStep
);

or you can use the cv::ml::SVM method cv::ml::SVM::getDefaultGrid(int). In this latter case, you need to tell getDefaultGrid() what parameter you would like a grid for. This is because the default grid is different for each argument. The valid values you can pass to getDefaultGrid are cv::ml::SVM::C, cv::ml::SVM::GAMMA, cv::ml::SVM::P, cv::ml::SVM::NU, cv::ml::SVM::COEF, or cv::ml::SVM::DEGREE. If you just want to create the grids yourself, you can do that as well, of course. Note that the argument logStep is interpreted as a multiplicative scale, so if you set it (for example) to 2.0, each value tried will be double the previous value until maxVal is reached (or exceeded).

Finally, now that you have trained your classifier, you can make predictions using the cv::ml::StatModel::predict() interface as usual:

float cv::ml::SVM::predict(
  InputArray  samples,
  OutputArray results = noArray(),
  int         flags   = 0
) const;

As usual, the predict() method expects one or more samples, one per row, and returns the resulting predictions. In the case of one-class or two-class classification it’s possible to retrieve the computed value instead of the chosen class label. In order to do that, pass the cv::ml::StatModel::RAW_OUTPUT flag. The returned signed value will correlate with the distance between sample and the decision surface; this is what you want in two class classification problems. Otherwise, a class label will be returned (for multiclass). In the case of regression, the return value will be the estimated value of the function, regardless of the flag.

Additional members of cv::ml::SVM

The cv::ml::SVM object also provides a few utility functions, which allow you to get at the data in the object. This includes data you put in at training time, but also includes useful things like the computed support vectors as well as the default grids. The available functions are:

// get all the support vectors
//
cv::Mat cv::ml::SVM::getSupportVectors()  const;

// get uncompressed support vectors as found by the training procedure
//
cv::Mat cv::ml::SVM::getUncompressedSupportVectors() const;

// get the i-th decision function (out of n*(n-1)/2 in the case of n-class problem
//
double cv::ml::SVM::getDecisionFunction(
  int             i,
  cv::OutputArray alpha,
  cv::OutputArray svidx
) const;

The method cv::ml::SVM::getSupportVectors() gives you all the support vectors used to compute the decision hyperplane or hypersurface. In the case of linear SVM, all the support vectors for each decision plane can be compressed into a single vector that will basically describe the separating hyperplane. That’s why linear SVM is super-fast at the prediction. However, users may be interested in looking at the original support vectors, which can be accessed via cv::ml::SVM::getUncompressedSupportVectors(). In the case of nonlinear SVM, uncompressed and compressed support vectors are the same thing. Then it’s possible to get access to each decision hyperplane or hypersurface. We do so using the method cv::ml::SVM::getDecisionFunction. In the case of regression, or one-class or two-class classification, there will be just one decision function, and so i=0. In the case of N-class classification, there will be N*(N-1)/2 decision functions, and so 0<=i<N*(N-1)/2. The method returns the coefficients for the support vectors used in the particular function, the indices of the support vectors (within the matrix returned by cv::ml::SVM::getSupportVectors()), and the value b (see the preceding formulas), which is added to the weighted sum before the decision is made.

In addition to the training and prediction methods, cv::ml::SVM supports the usual save(), load(), and clear() methods.