The term-document matrix thus prepared contains words from each of the tweets (post the cleanup and transformations) as rows, while columns represent the tweet themselves.

As a quick check, let us see which of the words are most frequently used in our dataset. Let the threshold be set to 30 occurrences or more. We use the apply utility to iterate each term in our term-document matrix and sum its occurrences. The function helps us filter out the terms that have appeared 30 times or more.

# Terms occuring in more than 30 times
> which(apply(twtrTermDocMatrix,1,sum)>=30)

The result will be as shown in the following screenshot:

Terms with 30 or more occurrences across tweets

As the preceding screenshot shows, words such as healthy, inspire, and positivity feature in the list of words with 30 or more occurrences. We all have a lot in common when it comes to yearly goals, no?

The preceding manipulation was a quick check to see if we really have tweets that help us find out something interesting about New Year's resolutions. Let us now take a formal approach and identify frequent terms in our data set. We will also try and present the information in a creative yet easy-to-understand representation. To get the most frequent terms in our data set, we use the function findFreqTerms from the tm package again. This function provides us an abstraction over quick hacks, such as the one we previously used. findFreqTerms also lets us set a minimum and maximum threshold for term frequencies. For our case, we will only mention the lower bound and see the results:

# print the frequent terms from termdocmatrix
> (frequentTerms<-findFreqTerms(twtrTermDocMatrix,lowfreq = 10))

The results look something like the following screenshot:

We get about 107 terms with a minimum occurrence of 10. If you look carefully, the terms we saw with frequencies of at least 30 also appear in this list, and rightly so.

Now that we are certain that there are terms/words with occurrences of more than 10, let us create a data frame and plot the terms versus their frequencies as we decided previously. We use the rowSums function to calculate the total occurrence of each term/word. We then pick a subset of terms which have more than 10 occurrences and plot them using ggplot:

# calculate frequency of each term
term.freq <- rowSums(as.matrix(twtrTermDocMatrix))

# picking only a subset
subsetterm.freq <- subset(term.freq, term.freq >= 10)


# create data frame from subset of terms
frequentTermsSubsetDF <- data.frame(term = names(subsetterm.freq), freq = subsetterm.freq)

# create data frame with all terms
frequentTermsDF <- data.frame(term = names(term.freq), freq = term.freq)

# sort by subset DataFrame frequency
frequentTermsSubsetDF <- frequentTermsSubsetDF[with(frequentTermsSubsetDF, order(-frequentTermsSubsetDF$freq)), ]

# sort by complete DataFrame frequency
frequentTermsDF <- frequentTermsDF[with(frequentTermsDF, order(-frequentTermsDF$freq)), ]

# words by frequency from subset data frame
ggplot(frequentTermsSubsetDF, aes(x = reorder(term,freq), y = freq)) + geom_bar(stat = "identity") +xlab("Terms") + ylab("Frequency") + coord_flip()

The preceding piece of code generates the following frequency graph:

Upon analyzing the preceding graph, we can quickly get some interesting points:

Though the preceding graph gives us quite a lot of information regarding the words and their frequencies in a nice layout, it still doesn't show us the full picture. Remember that we crafted a subset of items from our data set before generating this graph? We did that on purpose, otherwise the graph would have become too long and words with lesser frequencies would clutter the whole thing. Another point which this graph misses out is the relative difference in the frequencies.

If our aim is to see the relative difference between the frequencies, we need a different visualization altogether. Here comes word cloud to the rescue. Using the wordcloud library, we can easily generate word clouds from a dataframe using a one liner:

# wordcloud
> wordcloud(words=frequentTermsDF$term, freq=frequentTermsDF$freq,random.order=FALSE)

The wordcloud using the complete data frame looks something like this:

The preceding word cloud renders words in decreasing order of frequency. The size of each word emphasizes its frequency. You can play around with the wordcloud function to generate some interesting visualizations or even art!

A lot of words appear in the preceding graphs, but isn't it rather interesting to see the word trillionaire pop up in the top 10? What could be the reason for it? Was it a spam post by a bot, or a tweet by some celebrity that went viral, or something completely different altogether? Let's check out the top tweet in this list and see if it contains the word trillionaire:

# top retweets
> head(subset(trendingTweets.df$text, grepl("trillionaire",trendingTweets.df$text) ),n=1)

The following screenshot is what you get:

It turns out that our hunch was right. It was a New Year resolution tweet by a celebrity that went viral. A quick search on Twitter reveals the tweet:

Image source: https://twitter.com/mishacollins?lang=en

A bit further searching reveals Misha Collins is a famous actor from the television series Supernatural. We can also see that the above resolution was retweeted a staggering 5k times! It's interesting to note that the number of likes is 14k, outnumbering the retweets. Can we infer that tweeple prefer likes/hearts to retweets? It can also be seen that words such as mom, learn, trillionaire, elected, and President all occur as most frequent words without a doubt. Indirectly, we can also infer that Supernatural has a huge fan following on Twitter and that Castiel (Misha's role in the TV series) is a popular character from the show. A bit of a surprise is his resolution to learn to crochet, no?

Moving on from supernatural stuff, let us go back to the fitness debate. Fitness is important to most of us. Activities such as exercising or hitting the gym see a surge during the initial months/weeks of the year. Let's see how health-conscious our friends on Twitter are!

Since a lot of words such as health, diet, fitness, gym, and so on point towards a healthy lifestyle, let us try and find words associated with the word fitness itself. findAssocs is a handy function which helps us find words from a term-document matrix that have at least a specified level of correlation to a given word. We will use the output from this function to prepare a term-association (correlation) graph using ggplot. The process is similar to how we prepared the preceding frequency graph:

# Associatons
(fitness.associations <- findAssocs(twtrTermDocMatrix,"fitness",0.25))

fitnessTerm.freq <- rowSums(as.matrix(fitness.associations$fitness))

fitnessDF <- data.frame(term=names(fitnessTerm.freq),freq=fitnessTerm.freq)

fitnessDF <- fitnessDF[with(fitnessDF, order(-fitnessDF$freq)), ]
ggplot(fitnessDF,aes(x=reorder(term,freq),y=freq))
+geom_bar(stat = "identity") +xlab("Terms")
+ ylab("Associations")
+ coord_flip()

The words most closely correlated to the word fitness are as follows:

The same data is more readable in graphical form, as follows:

As evident from the preceding graph, terms such as lossweight, workout, getfit, and so on. prove our point that tweeple are as concerned about health as we are. It is interesting to note the occurrence of the term yogavideos in this list. It looks like yoga is catching up with other techniques of staying fit in 2016. There's meditation on the list too.

So far, we have dealt with the visible components of a tweet, such as the text, retweet counts, and so on, and we were able to extract many interesting insights. Let us take out our precision tools and dig deeper into our data.

As mentioned a couple times in the above sections, a tweet has far more information than what meets the eye. One such piece of information is about the source of the tweet. Twitter was born of the SMS era, and many of its characteristics, such as the 140 character word limit, are reminiscent of that era. It would be interesting to see how tweeple use Twitter, that is, what devices are used to access and post on Twitter frequently. Though the world has moved a long way from the SMS era, mobile phones are ubiquitous. To get this information, we will make use of the attribute tweetSource from our dataframe trendingTweets.df. We created this additional attribute from the statusSource attribute already existing in the tweet object (see the beginning of this section for a quick recap).

We shall use a subset of the data frame trendingTweets.df based upon retweet counts for the sake of clarity. We will use ggplot again to visualize our results.

# Source by retweet count
trendingTweetsSubset.df <- subset(trendingTweets.df, trendingTweets.df$retweetCount >= 5000 )

ggplot(trendingTweetsSubset.df, aes(x =tweetSource, y =retweetCount/100)) + geom_bar(stat = "identity") +xlab("Source") + ylab("Retweet Count")

The following plot is your result:

Without a doubt, the iPhone is the most preferred device, followed by Android and the Web. It is interesting to see that people use the Web/website to retweet more than the iPad! Windows Phone clearly has some serious issues to tackle here. Can we also infer that the iPhone is the preferred device amongst tweeples? Or does the iPhone provide a better experience than any other device for Twitter? Or we could even go deeper and say that Twitter on iPhone has an easier-to-access "retweets" button than any other device. Inferences such as these and many more, require a bit more digging than this, but all of this has a lot of knowledge/potential that could be used by managements, UX teams, and so on to improve and change things around.

We have seen clustering and classification in previous chapters (see Chapter 2, Let's Help Machines Learn) and uncovered some interesting facts about the data at hand. For our current use case, even though our tweets are all related to 2016 resolutions, we can never be sure of the kinds of resolutions tweeple make. This makes it a very apt use case for hierarchical clustering. Unlike k-means or other clustering algorithms that require a preset number of clusters before computation, hierarchical clustering algorithms work independently of it.

Let us take this opportunity to understand hierarchical clustering before we apply it to our data. Hierarchical clustering, like any other clustering algorithm, helps us group similar items together. The exact details for this algorithm in general can be explained as follows:

The merge and compute steps are repeated until we are left with a single cluster containing all n items. As the name suggests, this algorithm generates a hierarchical structure with the leaves denoting individual elements as clusters combined based upon similarity/distance as we go toward the root of the tree. The output tree is generally referred to as a dendrogram.

The merge step is where variations of this algorithm exist. There are several ways in which the closest clusters could be identified. From simple methods, such as single-link, which consider the shortest distance between any two elements of the two clusters in consideration as the distance measure, to complex ones such as Ward's method which uses variance to find the most compact clusters, there are several methods that could be employed depending upon the use case.

Coming back to the Twitter world, let us use hierarchical clustering to see which terms/tweets are the closest. For our current use case, we will use the single method for our merge criteria. You may try out different algorithms and observe the differences.

To perform hierarchical clustering, we first treat our dataset to remove sparse terms for the sake of clarity. For this, the removeSparseTerms function helps us remove rows of data that have sparsity below a specified limit. We then use the hclust utility to form clusters. The output of this utility is directly plottable. Let us write some code for this:

# remove sparse terms
twtrTermDocMatrix2 <- removeSparseTerms(twtrTermDocMatrix, sparse = 0.98)

tweet_matrix <- as.matrix(twtrTermDocMatrix2)

# cluster terms
distMatrix <- dist(scale(tweet_matrix))

fit <- hclust(distMatrix,method="single")
plot(fit)

The output dendrogram is amazingly simple to understand:

If you observe the second cluster from right, it contains terms trillionaire, elected, mom, call, and so on. Mapping back to the top retweeted tweet from Mischa Collins, all these terms are mentioned in that single tweet and our algorithm has rightly clustered them together. Smart, isn't it? As a small exercise, observe other clusters and see how the terms occur in the tweets that contain them. One important observation to make here is that the dendrogram correctly maps all frequent terms under a single root, which reaffirms that all these terms point to our central theme of 2016 resolutions!

So far, our analysis has been about tweets related to New Year's resolutions from users across the world. We have analyzed tweets related to a topic of our choice. Ignoring spam and other noisy tweets, more or less, our data conformed to a single topic. The topic itself constituted a group of words (such as health, trillionaire, fitness, diet, mom, and so on) which broadly describe different resolutions. To broaden our scope of analysis and to discover even more insights, let us touch upon the concept of topic modeling.

Topic modeling is a process of discovering patterns in a corpus of unlabeled text that represents the gist of the corpus. A topic itself may be described as a group of words that occur together to describe a large body of text.

Another definition tweeted during one of the conferences on topic modeling:

Image source: https://twitter.com/footnotesrising/status/264823621799780353

The aim of topic modeling is to automatically identify the underlying theme of a corpus and thus be useful in applications that require information retrieval based on a theme but in absence of known keywords (the exact opposite of our current usage of search engines). For example, wouldn't it be amazing to learn about relations between two countries from a newspaper's archive by using the theme relations between country one and country two rather than searching for a keyword and then following link after link. Please note that following links to discover information is equally powerful, but it leaves a lot to be desired.

One of the ways to perform topic modeling is through Latent Dirichlet Allocation (LDA); it is one of the most powerful and widely used models.

LDA was presented by David M Blie in his paper Introduction to Probabilistic Topic Models in 2003. LDA, as his paper says, can be defined as a generative model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data is similar. LDA works upon the assumption that documents exhibit multiple topics.

LDA is a probabilistic model and the mathematics of it are fairly involved and beyond the scope of this book. In a nonmathematical way, LDA can be explained as a model/process that helps identify the topics that have resulted in the generation of a collection of documents.

For our purpose/use case, we can assume LDA as a model/process which helps us to identify the underlying (hidden/latent) topics from a corpus of unlabeled text. Luckily, R abstracts most of the mathematical details in the form of a library called topicmodels.

For the purpose of topic modeling, we shall use a new set of tweets. The International Space Station (ISS) has multiple Twitter handles, and one of them is @ISS_Research, which particularly caters to research related tweets from the ISS. Let us explore what @ISS_Research is up to these days by analyzing the tweets from its timeline. We will analyze these tweets to identify the underlying topics of research at the ISS. For this purpose, we will use the same process to extract tweets and perform transformations/cleanup as we have done before. The following snippet of code does this:

# set user handle
atISS <- getUser("ISS_Research")

# extract iss_research tweets
tweets <- userTimeline(atISS, n = 1000)

tweets.df=twListToDF(tweets)

tweets.df$text <- sapply(tweets.df$text,function(x) iconv(x,to='UTF-8'))

#Document Term Matrix
twtrDTM <- DocumentTermMatrix(twtrCorpus, control = list(minWordLength = 1))

Please note that the preceding snippet prepares a document-term matrix, unlike last time where we prepared a term-document matrix.

Once we have tweets in the required format, the LDA utility from the topicmodels package helps us uncover the hidden topics/patterns. The LDA utility requires the number of topics as input along with the document-term matrix. We will try eight topics for now. The following code uses LDA to extract six terms for each of the eight topics:

#topic modeling

# find 8 topics
ldaTopics <- LDA(twtrDTM, k = 8) 

#first 6 terms of every topic
ldaTerms <- terms(ldaTopics, 6) 

# concatenate terms
(ldaTerms <- apply(ldaTerms, MARGIN = 2, paste, collapse = ", "))

The list of topics generated using LDA is as follows:

A visual representation would be easier to understand. We can make use of qplot to quickly plot the topics across time on an area chart, as follows:

# first topic identified for every tweet
firstTopic <- topics(ldaTopics, 1)

topics <- data.frame(date=as.Date(tweets.df$created), firstTopic)

qplot(date, ..count.., data=topics, geom="density",fill=ldaTerms[firstTopic], position="stack")+scale_fill_grey()

The generated chart looks like the following screenshot:

Let us now analyze the outputs. The list of terms per topic generated by LDA seems to give us a nice insight into the kind of work/research going on at the ISS. Terms such as mars, microgravity, flower, Cygnus, and so on tell us about the main areas of research or at least the topics about which scientists/astronauts on the ISS are talking. Terms such as stationcdrkelly and astrotimpeake look more like Twitter handles.

The qplot output adds the time dimension to our plain list of topics. Analyzing topics across the time dimension helps us understand when a particular research topic was discussed or when something amazing was announced. Topic two in the list, or the fourth one from the top in the graph legend comprises the word flower. Since scientists were successful in blooming some orange flowers in space recently, the above graph helps us get an idea that the news first broke on Twitter on/around 15^th January. A quick look on Twitter/news websites confirms that the news broke by tweet on 18^th January 2016…close enough!

So, finally we learnt about topic modeling using LDA on data from the ISS and found what amazing things scientists and astronauts are doing up there in outer space.