In this section, we will use Jane Austen's bestselling novel Pride and Prejudice, published in 1813, for our textual data preprocessing analysis. In R, we will use the tidytext package by Hadley Wickham to perform tokenization, stop word removal, sentiment extraction using predefined sentiment lexicons, term frequency - inverse document frequency (tf-idf) matrix creation, and to understand pairwise correlations among n-grams.

In this section, instead of storing text as a string or a corpus or a document term matrix (DTM), we process them into a tabular format of one token per row.