This chapter is all about social media. Though it's not directly related to NLTK / NLP, social data is also a very rich source of unstructured text data. So, as NLP enthusiasts, we should have the skill to play with social data. We will try to explore how to gather relevant data from some of the most popular social media platforms. We will cover how to gather data from Twitter, Facebook, and so on using Python APIs. We will explore some of the most common use cases in the context of social media mining, such as trending topics, sentiment analysis, and so on.
You already learned a lot of topics under the concepts of natural language processing and machine learning in the last few chapters. We will try to build some of the applications around social data in this chapter. We will also provide you with some of the best practices to deal with social data, and look at social data from the context of graph visualization.
There is a graph that underlies social media and most of the graph-based problems can be formulated as information flow problems and finding out the busiest node in the graph. Some of the problems such as trending topics, influencer detection, and sentiment analysis are examples of these. Let's take some of these use cases, and build some cool applications around these social networks.
By the end of this chapter,:
- You should be able to collect data from any social media using APIs.
- You will also learn to formulate the data in a structured format and how to build some amazing applications.
- Lastly, we will be able to visualize and gain meaningful insight out of social media.
The most important objective of this chapter is to gather data across some of the most common social networks. We will look mainly at Twitter and Facebook and try to give you enough details about the API and how to effectively use them to get relevant data. We will also talk about the data dictionary for scrapped data, and how we can build some cool apps using some of the stuff we learned so far.
We will start with one of the most popular and open social media that is completely public. This means that practically, you can gather entire Twitter stream, which is payable, while you can capture one percent of the stream for free. In the context of business, Twitter is a very rich resource of information such as public sentiments and emerging topics.
Let's get directly to face the main challenge of getting the tweets relevant to your use case.
There are more than 10 Python libraries there. So pick and choose the one you like. I generally use Tweepy and we will use it to run the examples in this book. Most of the libraries are wrappers around the Twitter API, and the parameters and signatures of all these are roughly the same.
The simplest way to install Tweepy is to install it using pip:
$ pip install tweepy
To get Tweepy to work, you have to create a developer account with Twitter and get the access tokens for your application. Once you complete this, you will get your credentials and below these, the keys. Go through https://apps.twitter.com/app/new for registration and access tokens. The following snapshot shows the access tokens:
We will start with a very simple example to gather data using Twitter's streaming API. We are using Tweepy to capture the Twitter stream to gather all the tweets related to the given keywords:
tweetdump.py >>>from tweepy.streaming import StreamListener >>>from tweepy import OAuthHandler >>>from tweepy import Stream >>>import sys >>>consumer_key = 'ABCD012XXXXXXXXx' >>>consumer_secret = 'xyz123xxxxxxxxxxxxx' >>>access_token = '000000-ABCDXXXXXXXXXXX' >>>access_token_secret ='XXXXXXXXXgaw2KYz0VcqCO0F3U4' >>>class StdOutListener(StreamListener): >>> def on_data(self, data): >>> with open(sys.argv[1],'a') as tf: >>> tf.write(data) >>> return >>> def on_error(self, status): >>> print(status) >>>if __name__ == '__main__': >>> l = StdOutListener() >>> auth = OAuthHandler(consumer_key, consumer_secret) >>> auth.set_access_token(access_token, access_token_secret) >>> stream = Stream(auth, l) >>> stream.filter(track=['Apple watch'])
In the preceding code, we used the same code given in the example of Tweepy, with a little modification. This is an example where we use the streaming API of Twitter, where we track Apple Watch. Twitter's streaming API provides you the facility of conducting a search on the actual Twitter stream and you can consume a maximum of one percent of the stream using this API.
In the preceding code, the main parts that you need to understand are the first and last four lines. In the initial lines, we are specifying the access tokens and other keys that we generated in the previous section. In the last four lines, we create a listener to consume the stream. In the last line, we use stream.filter to filter Twitter for keywords that we have put in the track. We can specify multiple keywords here. This will result in all the tweets that contain the term Apple Watch for our running example.
In the following example, we will load the tweets we have collected, and have a look at the tweet structure and how to extract meaningful information from it. A typical tweet JSON structure looks similar to:
{ "created_at":"Wed May 13 04:51:24 +0000 2015", "id":598349803924369408, "id_str":"598349803924369408", "text":"Google launches its first Apple Watch app with News & Weather http://t.co/o1XMBmhnH2", "source":"u003ca href="http://ifttt.com" rel="nofollow"u003eIFTTTu003c/au003e", "truncated":false, "in_reply_to_status_id":null, "user":{ "id":1461337266, "id_str":"1461337266", "name":"vestihitech u0430u0432u0442u043eu043cu0430u0442", "screen_name":"vestihitecha", "location":"", "followers_count":20, "friends_count":1, "listed_count":4, ""statuses_count":7442, "created_at":"Mon May 27 05:51:27 +0000 2013", "utc_offset":14400, }, , "geo":{ "latitude" : 51.4514285, "longitude"=-0.99 } "place":"Reading, UK", "contributors":null, "retweet_count":0, "favorite_count":0, "entities":{ "hashtags":["apple watch", "google" ], "trends":[ ], "urls":[ { "url":"http://t.co/o1XMBmhnH2", "expanded_url":"http://ift.tt/1HfqhCe", "display_url":"ift.tt/1HfqhCe", "indices":[ 66, 88 ] } ], "user_mentions":[ ], "symbols":[ ] }, "favorited":false, "retweeted":false, "possibly_sensitive":false, "filter_level":"low", "lang":"en", "timestamp_ms":"1431492684714" } ]