Reading the CSV File

Using a read_csv function from the pandas library, we read the Reviews.csv file into a food_review data frame and print the top rows (Figure 5-3):

import pandas as pd

food_review = pd.read_csv("Reviews.csv")

food_review.head()

As can be seen, the CSV contains columns like ProductID, UserID, Product Rating, Time, Summary, and Text of the review. The file contains almost 500K reviews for various products. Let’s sample some reviews to process.

Sampling

Using the sample function from the pandas data frame, let’s randomly pick the text of 1000 reviews and print the top rows (see Figure 5-4):

food_review_text = pd.DataFrame(food_review["Text"])

food_review_text_1k = food_review_text.sample(n= 1000,random_state = 123)

food_review_text_1k.head()

Tokenization Using NLTK

As discussed, NLTK offers many features for processing textual data. The first step in processing text data is to separate a sentence into individual words. This process is called tokenization. We will use the NLTK’s word_tokenize function to create a column in the food_review_text_1k data frame we created above and print the top six rows to see the output of tokenize (Figure 5-5):

food_review_text_1k['tokenized_reviews'] = food_review_text_1k['Text'].apply(nltk.word_tokenize)

food_review_text_1k.head()

Normalization Using NLTK

In many natural language tasks, we often deal with the root form of the words. For example, for the words “baking” and “baked,” the root word is “bake.” This process of extracting the root word is called stemming or normalization. NLTK provides two functions implementing the stemming algorithm. The first is the Porter Stemming algorithm, and the second is the Lancaster stemmer.

Noun Phrase Chunking Using Regular Expressions

Above you saw the tokens as a fundamental unit in any NLP processing. Since in natural language, a group of tokens combined often reveals the meaning or represents a concept, we create chunks. Multi-token sequences are created by segmenting using a process called chunking. In Figure 5-7, the smaller boxes show word-level tokenization and the larger boxes shows multi-token sequences, also called higher-level chunks. Such chunks are created using regular expressions or by using the n-gram (more on this in later sections) method. Chunking is essential for entity recognition, which we will shortly explore.

You can see many NPs such as “the summer” and “age” where “the summer” is not a single word token. Above you see that the POS is in a tree representation. Another way of representing the chunk structures is by using tags. The IOB tag representation is a general standard. In this scheme, each token is represented as I (Inside), O (Outside), and B (Begin). Chunk tag B represents the beginning of a chunk. Subsequent tokens within a chunk are tagged I and all other tokens are tagged O. Figure 5-8 provides one example of an IOB tag representation.

Named Entity Recognition

Once we have the POS of the text, we can extract the named entities. Named entities are definite noun phrases that refer to specific individuals such as ORGANIZATION and PERSON. Some other entities are LOCATION, DATE, TIME, MONEY, PERCENT, FACILITY, and GPE. The facility is any human-made artifact in the architecture and civil engineering domain, such as Taj Mahal or Empire State Building. GPE means geopolitical entities such as city, state, and country. We can extract all these entities using the ne_chunk() method in the nltk library.

POS Tagging

Dependency Parsing

Dependency Tree

spaCy provides a method called displayCy for visualization. We can draw the dependency tree of a given sentence using displaCy (see Figures 5-9, 5-10, and 5-11).

import spacy

from spacy import displacy

nlp = spacy.load("en_core_web_sm")

doc = nlp(u"My English Bulldog Larry had skin allergies the summer we got him at age 3")

displacy.render(doc, style="dep")

From the dependency trees, you can see that there are two compound word pairs, “English Bulldog” and “skin allergies,” and NUM “3” is the modifier of “age.” You can also see “summer” as the noun phrase as an adverbial modifier (npadvmod) to the token “had.” You can also observe many direct objects (dobj) of a verb phrase, which is a noun phrase, like (got, him) and (had, allergies) and object of a preposition (pobj) like (at, age). A detailed explanation of the relationships in a dependency tree can be found here: https://nlp.stanford.edu/software/dependencies_manual.pdf .

Chunking

Named Entity Recognition

spaCy has an accuracy of 85.85% in named entity recognition (NER) tasks. The en_core_web_sm model provides the function ents, which provides the entities. The model is trained on the OntoNotes dataset, which can be found at https://catalog.ldc.upenn.edu/LDC2013T19 .

We can also visualize the entities using the displacy method (shown in Figure 5-12):

import spacy

from spacy import displacy

from pathlib import Path

text = "I found these crisps at our local WalMart & figured I would give them a try. They were so yummy I may never go back to regular chips, not that I was a big chip fan anyway. The only problem is I can eat the entire bag in one sitting. I give these crisps a big thumbs up!"

nlp = spacy.load("en_core_web_sm")

doc = nlp(text)

svg = displacy.serve(doc, style="ent")

output_path = Path("images/sentence_ne.svg")

output_path.open("w", encoding="utf-8").write(svg)

Pattern-Based Search

In the search span, if we want to find the word “Walmart,” we define this using the matcher.add method and pass pattern as the argument to the method.

Searching for Entity

Training a Custom NLP Model

Tokenizing

Part-of-Speech Tagging

POS can be extracted using the method pos_tag in the

Named Entity Recognition

Constituency Parsing

Constituency parsing extracts a constituency-based parse tree from a given sentence that is representative of the syntactic structure according to a phase structure grammar. See Figure 5-13 for a simple example.

Dependency Parsing

POS Tags and Noun Phrase

Spelling Correction

Spelling correction is an exciting feature of TextBlob, which is not provided in the other libraries described in this chapter. The implementation is based on a simple technique provide by Peter Norvig, which is only 70% accurate. The method correct in TextBlob provides this implementation.

Machine Translation

TextBlob for Translation

As shown in the example above, we use TextBlob for machine translation so non-French readers can understand the text we process.

POS and Dependency Relations

Its performance of French POS and dependency relation is entirely accurate. It can identify almost all the VERBS, NOUNS, ADJ, PROPN, and many other tags. Next, let’s see how it performs on the entity recognition task.

Named Entity Recognition

Noun Phrases

Polarity

A float value within the range [-1.0, 1.0]. This scoring uses a corpus of positive, negative, and neutral words (which is called polarity) and detects the presence of a word in any of the three categories. In a simple example, the presence of a positive word is given a score of 1, -1 for negative, and 0 for neutral. We define polarity of a sentance as the average score, i.e., the sum of the scores of each word divided by the total number of words in the sentance.

If the value is less than 0, the sentiment of the sentence is negative and if it is greater than 0, it is positive; otherwise, it’s neutral.

Subjectivity

A float value within the range [0.0, 1.0]. A perfect score of 1 means “very subjective.” Unlike polarity, which reveals the sentiment of the sentence, subjectivity does not express any sentiment. The score tends to 1 if the sentence contains some personal views or beliefs. The final score of the entire sentence is calculated by assigning each word on subjectivity score and applying the averaging; the same way as polarity.

The TextBlob library internally calls the pattern library to calculate the polarity and subjectivity of a sentence. The pattern library uses SentiWordNet, which is a lexical resource for opinion mining, with polarity and subjectivity scores for all WordNet synsets. Here is the link to the SentiWordNet: https://github.com/aesuli/sentiwordnet .

Word2Vec

Word2Vec is a machine learning model (trained with a vast vocabulary of words using the neural network) that produces word embeddings, which are vector representations of words in the vocabulary. Word2vec models are trained to construct the linguistic context of words. We will see some examples in Python using the gensim library to understand what linguistic context means. Figure 5-14 shows the neural network architecture for training the Word2Vec model.

A skip-gram neural network model for Word2Vec computes the probability for every word in the vocabulary of being the “nearby word” that we select. Proximity or nearness of words can be defined by a parameter called window size. Figure 5-14 shows the possible pair of words for training a neural network with window size of 2.

In the input sentence, “Building an enterprise chatbot that can converse like humans” is broken into words and with a window size of 2, we take two words each from left and right of the input word. So, if the input word is “chatbot,” the output probability of the word “enterprise” will be high because of its proximity to the word “chatbot” in the window size of 2. This is only one example sentence. In a given vocabulary, we will have thousands of such sentences; the neural network will learn statistics from the number of times each pairing shows up. So, if we feed many more training samples like the one shown in Figure 5-14, it will figure out how likely the words “chatbot” and “enterprise” are going to appear together.

Neural Network Architecture

The input vector to the neural network is a one-hot vector representing the input word “chatbot,” by storing 1 in the ith position of the vector and 0 in all other positions, where 0 ≤ i ≤ n and n is the size of the vocabulary (set of all the unique words)

In the hidden layer, each word vector of size n is multiplied with a feature vector of size, let’s say 1000. When the training starts, the feature vector of size 1000 are all assigned a random value. The result of the multiplication will select the corresponding row in the n x 1000 matrix where the one-hot vector has a value of 1.

Finally, in the output layer, an activation function like softmax is applied to shrink the output value to be between 0 and 1. The following equation represents the softmax function, where K is the size of the input vector:

So, if the input vector representing “chatbot” is multiplied with the output vector represented by “enterprise,” the softmax function will be close to 1 because in our vocabulary, both the words appeared together very frequently.

Neural networks train the network and update the edge weights over many iterations of training. The final set of weights represents learnings. Figure 5-15 shows the neural network architecture to train a Word2Vec model.

Using the Word2Vec Pretrained Model

In the following code, we use the pretrained Word2Vec model from a favorite Python library called gensim. Word2Vec models provide a vector representation of words that make various natural language tasks possible, such as identifying similar words, finding synonyms, word arithmetic, and many more. The most popular Word2Vec models are GloVe, CBOW, and skip-gram. In this section, we will use all three models to perform various tasks of NLU.

In the demo, we use the model to perform many syntactic/semantic NLU word tasks.

Performing Out-of-the-Box Tasks Using a Pretrained Model

One of the useful features of gensim is that it offers several pretrained word vectors from gensim-data. Apart from Word2Vec, it also provides GloVe, another robust unsupervised learning algorithm for finding word vectors. The following code downloads a glove-wiki-gigaword-100 word vector from gensim-data and performs some out-of-the-box tasks.

Step 2: Compute the nearest neighbors. As you have seen, word vectors contain an array of numbers representing a word. Now it becomes possible to perform mathematical computations on the vectors. For example, we can compute Euclidean or cosine similarities between any two-word vectors. There are some interesting results that we obtain as a result. The following code shows some of the outcomes.

Figure 5-14 shows an example of how the input data for training the neural network was created by shifting the window of a size 2. In the following example, you will see that “apple” on the Internet is no longer fruit; it has become synonymous with the Apple Corporation and shows many companies like it when we compute a word similar to “apple.” The reason for such similarity is because of the vocabulary used for training, which in this case is a Wikipedia dump of close to 6 billion uncased tokens. More such pretrained models are available at https://github.com/RaRe-Technologies/gensim-data .

Step 3: Identify linear substructures. The relatedness of two words is easy to compute using the similarity or distance measure, whereas to capture the nuances in a word pair or sentences in a more qualitative way, we need operations. Let’s see the methods that the gensim package offers to accomplish this task.

fastText Word Representation Model

Similar to the examples discussed in this section, using either the skip-gram or CBOW model, various tasks can be performed. We can evaluate the performance to choose the best model for our final implementation.

Collection of Documents

Loading Libraries and Defining Stopwords

Removing Common Words and Tokenizing

Removing Words That Appear Infrequently

Now we see the words that occur more than once. For our example, it seems like there are not many words with more than one occurrence. We expect the model not to perform very well. However, let’s still go ahead with training the model.

Saving the Training Data as a Dictionary

Generating the Bag of Words

Training the Model Using LDA

Finally, using the bag-of-words dictionary of words, we train the latent Dirichlet allocation model. LDA is a generative statistical model where, given an input variable X and target variable Y, the model based on joint probability is X ∗ Y, P(X, Y). LDA is a favorite machine learning model widely used in topic modeling. Each document (in our example, each review) is a mixture of various topics, where each document is assigned a set of topics by LDA.

The gensim library provides a method called LsiModel() , which trains an LDA model. Latent semantic indexing (LSI) is used in the context of LDA’s application in information retrieval.

Note that a more accurate model would need plenty of data for training and perhaps many more interesting topics might evolve.

Loading the Library

Loading the File and Printing the Headlines

Read the ABC news dataset from a CSV file using the read_csv() method and print the top three news headlines. The dataset contains over 15 years of news headlines published by the Australian Broadcasting Corp. The dataset contains more than 1 million news headlines. The dataset is available for download from www.kaggle.com/therohk/million-headlines/data . See Figure 5-16.

#Reading input text file

Input_text = pd.read_csv('data/abcnews-date-text.csv')

Input_text.head(3)

Building a Text Model Using Markovify

Generating Random Headlines

Loading the Library

Tense

Negation

Interrogative

Complements

Modifiers

Prepositional Phrases

Coordinated Clauses

Subordinate Clauses

Main Method

Printing the Output

As you can see, SimpleNLG offers an easy-to-use syntax to generate grammatically correct sentences in English programmatically. Next, let’s dive into a deep learning model to generate the next words given a piece of text. Unlike SimpleNLG, we are not sure if we’ll get a grammatically correct sentence in such a deep learning model.

Loading the Library

Defining the Training Data

Data Preparation

Creating an RNN Architecture Using a LSTM Network

Defining the Generate Text Method

Training the RNN Model

Generating Text

Tokenizing and Cleaning the Text

Lemmatization

Preprocessing the Text Method for LDA

Reading the Training Data

Bag of Words

Training and Saving the Model

From the output above, it looks like topics 0 and 3 are about a “ginger flavor corn syrup” while topics 2 and 4 are not very clear on what they convey. Moreover, topic 1 talks about “tortilla chips.”

Predictions

Loading the NLTK Library and Downloading the Names Corpus

Loading the Male and Female Names

Common Names

Extract Features

Randomly Splitting into Train and Test

Training the Model

Model Prediction

Model Accuracy

Most Informative Features

Using the show_most_informative_features() method from the model, we can see which last letters from the names are essential for classifying the male and female names.

Loading Libraries

Reading the Dataset into the Categorized Corpus

Computing Word Frequency

Checking the Presence of Frequent Words

Training the Model

Most Informative Features

In this corpus, a review that mentions “not” is almost five times more likely to be negative than positive, while a review that mentions “good” is only about three times more likely to be negative than positive. Perhaps the negative-ness of the word “good” might stem from the customers with reviews of the nature, “the product is good but ...” where they may have one or two complain.

If we add more reviews to this corpus of positive and negative, the accuracy will start to improve.

Intent Classification

We need to classify a given query (input from the user) into an intent class. Once an intent class is identified, a chatbot can trigger the respective logic as a response to a user query. For example, if the query is “how is the weather today,” the intent classification should trigger the weather services API from within the chatbot and fetch the result.

Question Answering

Chatbots often need to understand the context of the conversation to answer a particular query from a user. The deeppavlov library provides a pretrained model trained on Stanford Question Answering Dataset (SQuAD) dataset, a reading comprehension dataset consisting of crowdsourced questions on a set of Wikipedia articles. More details on the dataset can be found at https://rajpurkar.github.io/SQuAD-explorer/ .

The main task the model trained on SQuAD dataset performs is to identify a given context and answer a question within the given context.