Once again, we simplified our task a bit, since we used only positive or negative tweets. That means, we assumed a perfect classifier that classified upfront whether the tweet contains a sentiment and forwarded that to our Naïve Bayes classifier.

So, how well do we perform if we also classify whether a tweet contains any sentiments at all? To find that out, let's first write a convenience function that returns a modified class array that provides a list of sentiments that we would like to interpret as positive:

def tweak_labels(Y, pos_sent_list):
    pos = Y==pos_sent_list[0]
    for sent_label in pos_sent_list[1:]:
        pos |= Y==sent_label
    
    Y = np.zeros(Y.shape[0])
    Y[pos] = 1
    return Y.astype(int)  

Note that we are talking about two different positives now. The sentiment of a tweet can be positive, which is to be distinguished from the class of the training data. If, for example, we want to find out how well we can separate tweets having a sentiment from neutral ones, we could do:

>>> X = X_orig
>>> Y = tweak_labels(Y_orig, ["positive", "negative"])  

In Y we now have 1 (positive class) for all tweets that are either positive or negative and 0 (negative class) for neutral and irrelevant ones:

>>> train_model(create_ngram_model, X, Y)
Mean acc=0.734    Mean P/R AUC=0.661  

Have a look at the following plot:

As expected, P/R AUC drops considerably, being only 66% now. The accuracy is still high, but that is only due to the fact that we have a highly imbalanced dataset. Out of 3,077 total tweets, only 839 are either positive or negative, which is about 27%. This means, if we create a classifier that always classifies a tweet as not containing any sentiments, we will already have an accuracy of 73%. This is another reason to always look at precision and recall if the training and test data is unbalanced.

So, how will the Naïve Bayes classifier perform on classifying positive tweets versus the rest, and negative tweets versus the rest? One word: poorly:

== Pos vs. rest == 
Mean acc=0.87  Mean P/R AUC=0.305 
== Neg vs. rest == 
Mean acc=0.852 Mean P/R AUC=0.49 

Pretty unusable if you ask me. Looking at the P/R curves in the following plots, we will also find no usable precision/recall trade-off, as we were able to do in the last chapter: