Since this is a supervised learning approach, we require labeled data for training and testing the performance of our algorithms. For the purpose of this chapter, we will utilize a labeled dataset from http://www.sentiment140.com/. It contains tweets labeled as 0, 2, and 4 for negative, neutral and positive sentiments, respectively. There are various attributes such as tweet ID, date, search query, username, and the tweet text, apart from the sentiment label. For our case we will be considering only the tweet text and its corresponding label.

Before we get into the algorithm-specific details, let us look into the labeled dataset and perform the initial steps of collecting and transforming our data into the required forms. We will make use of libraries such as caret and RTextTools for these steps.

As mentioned previously, the dataset contains polarities labeled as 0, 2, and 4 for negative, neutral, and positive. We will load the csv file in R and apply a quick transformation to change the labels to positive and negative. Once the polarities have been transformed into intelligible names, we will filter out the rows of data containing neutral sentiments. Also, we will keep only the columns for polarity and tweet text, and remove the rest.

# load labeled dataset
labeledDSFilePath = "labeled_tweets.csv"
labeledDataset = read.csv(labeledDSFilePath, header = FALSE)

# transform polarity labels
labeledDataset$V1 = sapply(labeledDataset$V1, 
    function(x) 
      if(x==4) 
        x <- "positive" 
      else if(x==0) 
        x<-"negative" 
      else x<- "none")

#select required columns only
requiredColumns <- c("V1","V6")

# extract only positive/negative labeled tweets 
tweets<-as.matrix(labeledDataset[labeledDataset$V1 
      %in% c("positive","negative")
      ,requiredColumns])

The tweets object is now available as a matrix with each row representing a tweet, and with columns referring to polarity and tweet text. Before we transform this matrix into the formats required by the classification algorithms, we need to split our data into training and testing datasets (see Chapter 2, Let's Help Machines Learn, for more on this). Since both the training and testing datasets should contain a good enough distribution of samples of all classes for the purposes of training and testing, we use the createDataPartition function available from the caret package. For our use case, we split our data into 70/30 training and testing datasets:

indexes <- createDataPartition(tweets[,1], p=0.7, list = FALSE)

train.data <- tweets[indexes,]
test.data <- tweets[-indexes,]

We perform a quick check to see how our data is split across the positive and negative classes in our original dataset, and the training and testing datasets. You will see the result in the following screenshot:

As we can see, createDataPartition has done a nice job of maintaining a similar sentiment distribution across the training and testing datasets.

Next in the line of transformations is the Document Term Matrix transformation. As we have seen in Chapter 7, Social Media Analysis – Analyzing Twitter Data, a document term matrix transforms a given dataset into rows representing the documents and columns of terms (words/sentences). Unlike the previous chapter, where we used the tm library's DocumentTermMatrix function for transformation and applied various transformations using tm_map, for the current use case we will use the create_matrix function from the RTextTools library. This function is an abstraction over tm's corresponding functions. We will also assign weights to each of the terms using tfidf as our feature. The create_matrix method also helps us take care of splitting sentences into words, stop words and number removal, and stemming them as well. Here's how you do it:

train.dtMatrix <- create_matrix(train.data[,2], 
                        language="english" , 
                        removeStopwords=TRUE, 
                        removeNumbers=TRUE,
                        stemWords=TRUE,
                        weighting = tm::weightTfIdf)
            

            
test.dtMatrix <- create_matrix(test.data[,2], 
                               language="english" , 
                               removeStopwords=TRUE, 
                               removeNumbers=TRUE,
                               stemWords=TRUE,
                               weighting = tm::weightTfIdf,
                               originalMatrix=train.dtMatrix)

test.data.size <- nrow(test.data)

>  trace("create_matrix",edit=T) 

Now that we have both the training and testing datasets in the DocumentTermMatrix format, we can proceed towards the classification algorithms and let our machines learn and build sentiment classifiers!

Support Vector Machines, or SVM as they are commonly known, are one of the most versatile classes of supervised learning algorithms for classification. An SVM builds a model in such a way that the data points belonging to different classes are separated by a clear gap, which is optimized such that the distance of separation is the maximum possible. The samples on the margins are called the support vectors, which are separated by a hyperplane (see Chapter 6, Credit Risk Detection and Prediction – Predictive Analytics for more details).

Since our current use case for sentiment analysis is also a binary (positive and negative) classification problem, SVM helps us build a model using the training dataset, which separates tweets into positive and negative sentiment classes, respectively.

We will use e1071 library's svm function to build a sentiment classifier. We start off with the default values for the SVM classifier available from the library and follow the same iterative procedure we did in Chapter 6, Credit Risk Detection and Prediction – Predictive Analytics, to finally arrive at the best classifier. The following snippet of code builds a sentiment classifier using the default values and then prints a confusion matrix, along with other statistics for evaluation, as shown in the following code snippet:

svm.model <- svm(train.dtMatrix, as.factor(train.data[,1]))

## view inital model details
summary(svm.model)

## predict and evaluate results
svm.predictions <- predict(svm.model, test.dtMatrix)

true.labels <- as.factor(test.data[,1])

confusionMatrix(data=svm.predictions, reference=true.labels, positive="positive")

The confusion matrix generated as follows shows that the classifier has just 50% accuracy, which is as bad as a coin toss, with no predictions for negative sentiments whatsoever! It seems like the classifier couldn't infer or learn much from the training dataset.

To build a better-performing model, we will now go under the hood and tweak some parameters. The svm implementation from e1071 provides us with a wonderful utility called tune to obtain the optimized values of hyperparameters using a grid search over the given parameter ranges:

## hyperparameter optimizations

# run grid search
cost.weights <- c(0.1, 10, 100)
gamma.weights <- c(0.01, 0.25, 0.5, 1)
tuning.results <- tune(svm, train.dtMatrix, as.factor(train.data[,1]), kernel="radial", 
                       ranges=list(cost=cost.weights, gamma=gamma.weights))

# view optimization results
print(tuning.results)

# plot results
plot(tuning.results, cex.main=0.6, cex.lab=0.8,xaxs="i", yaxs="i")

The parameter-tuning results in optimized values for hyperparameters cost and gamma as 10 and 0.01, respectively; the following plot confirms the same (darkest region corresponds to best values).

The following snippet of code uses the best model to predict and prepare a confusion matrix, as follows:

# get best model and evaluate predictions
svm.model.best = tuning.results$best.model

svm.predictions.best <- predict(svm.model.best, test.dtMatrix)

confusionMatrix(data=svm.predictions.best, reference=true.labels, positive="positive")

The following confusion matrix shows the predictions from a much improved model. From a mere 50% accuracy to a comfortable 80% and above is a good leap. Let us check the ROC curves for this model to confirm that the accuracy is indeed good enough:

To prepare the ROC curves, we will reuse our utility script performance_plot_utils.R from Chapter 6, Credit Risk Detection and Prediction – Predictive Analytics, and pass the predictions from the optimized model to it:

# plot best model evaluation metric curves
svm.predictions.best <- predict(svm.model.best, test.dtMatrix, decision.values = T)

svm.prediction.values <- attributes(svm.predictions.best)
$decision.values

predictions <- prediction(svm.prediction.values, true.labels)

par(mfrow=c(1,2))
plot.roc.curve(predictions, title.text="SVM ROC Curve")
plot.pr.curve(predictions, title.text="SVM Precision/Recall Curve")

The ROC curves generated using the preceding code snippet are as follows:

The ROC curves also confirm a well-learned model with an AUC of 0.89. We can therefore use this model to classify tweets into positive or negative classes. We encourage readers to try out ROC-based optimizations and observe if there are any further improvements in the model.

Supervised Machine Learning algorithms, in a nutshell, are about learning the underlying functions or patterns which help us predict accurately (within certain bounds) based on historic data. Over the course of this book, we have come across many such algorithms and, although R makes it easy to code and test these, it is worth mentioning that learning a highly accurate function or pattern is not an easy task. Building highly complex models leads us to issues of overfitting and underfitting, to name a few. Amidst all this confusion, it is to be noted that it is always easy to learn simple rules and functions.

For example, to classify an email as spam or not spam there are multiple rules which a machine learning algorithm would have to learn, rules such as:

And many more such rules. Given a training dataset, say T of labeled emails, a machine learning algorithm (specifically a classification algorithm) will generate a classifier, C, which is a hypothesis of an underlying function or pattern. We then use this classifier C to predict the labels for new emails.

On the other hand, an ensemble of classifiers is defined as a set of classifiers whose outputs are combined in some way to classify new examples. The main discovery in the field of machine learning-based ensembles is that ensembles perform much better than the individual classifiers they are made of.

A necessary and sufficient condition for ensembles to be better than their constituents is that they should be accurate and diverse. A classifier is termed accurate if its predictions are better than random guessing (see weak learners ). While two classifiers are termed as diverse if they make different errors on the same data points.

We can define a weak learner as a learner whose predictions and decisions are at least better than random guessing. Weak learners are also termed as base learners or meta learners.

The following block diagram visualizes the concept of ensemble classifiers:

As seen in the preceding block diagram, the training dataset is split into n datasets (the splitting or generation of such datasets is dependent upon the ensemble-ing methodology) upon which weak learners (the same or different weak learners, again, depends upon the ensemble methodology) build models. These models are then combined based on weighted or unweighted voting to prepare a final model, which is used for classification. The mathematical proofs of why ensembles work are fairly involved and beyond the scope of this book.

Boosting

There are various ways of constructing ensemble classifiers (or regressors) and boosting is one of them. Boosting came out as an answer by Robert Schapire in his pioneering paper in 1990 entitled The Strength of Weak Learnability, where he elegantly describes the boosting ensemble while answering questions posed by Kearns and Valiant in their paper published in 1989, which talks about multiple weak learners that can create a single strong learner.

Note

The Strength of Weak Learnability: http://www.cs.princeton.edu/~schapire/papers/strengthofweak.pdf.

Kearns and Valiant: Cryptographic limitations on learning Boolean Learning and finite automata: http://dl.acm.org/citation.cfm?id=73049

The original algorithm for boosting was revised by Freund and Schapire, and termed as AdaBoost or Adaptive Boosting. This algorithm was practically implementable and empirically improves generalization performance. The algorithm can be mathematically presented as follows:

Source: https://www.cs.princeton.edu/picasso/mats/schapire02boosting_schapire.pdf

Here:

• X is the training set
• Y is the label set
• D_t(i) is the weight distribution on training example i on iteration t
• h_t is the classifier obtained in iteration t
• α is the strength parameter or weight of h_t
• H is the final or combined classifier.

In simple words, boosting, in general, begins by initially assigning equal weights to all training examples. It then iterates over the hypothesis space to learn a hypothesis h_t on the weighted examples. After each such hypothesis is learned, the weights are adjusted in such a manner that the weights of the examples that are correctly classified are reduced. This update to weights helps weak learners, in coming iterations, to concentrate more on wrongly classified data points. Finally, each of the learned hypotheses is then passed through a weighted voting to come up with a final model, H.

Now that we have an overview of ensemble methods and boosting in general, let us use the boosting implementation available from the RTextTools library in R to classify tweets as positive or negative.

We will reuse the training-testing document term matrices train.dtMatrix and test.dtMatrix, and container objects train.container and test.container, which we created for the SVM-based classification.

For building a classifier based on a boosting ensemble, RTextTools provides an easy-to-use utility function called train_model. It uses LogitBoosting internally to build a classifier. We use 500 iterations for building our boosting ensemble.

boosting.model <- train_model(train.container, "BOOSTING"
            , maxitboost=500)
boosting.classify <- classify_model(test.container, boosting.model)

We then prepare a confusion matrix to see how our classifier performs on the test dataset.

predicted.labels <- boosting.classify[,1]
true.labels <- as.factor(test.data[,1])

confusionMatrix(data = predicted.labels, 
                reference = true.labels, 
                positive = "positive")

The following confusion matrix shows that our boosting-based classifier works with an accuracy of 78.5%, which is fairly good given the fact that we did not perform any performance tuning. Compare this to the initial iteration of SVM where we got a dismal accuracy of just over 50%.

As mentioned earlier, ensemble methods (specifically boosting) have improved generalized performance, that is, they help achieve close to 0 training errors without overfitting on the training data. To understand and evaluate our Boosting classifier on these parameters, we will use a model-evaluation technique called Cross-validation.

# Cross validation
N=10
set.seed(42)
cross_validate(train.container,N,"BOOSTING"
    , maxitboost=500)