In this chapter, we harvested data from Twitter. Once the data was acquired, we explored the information using Continuum.io's Blaze and Odo libraries. Spark SQL is an important module for interactive data exploration, analysis, and transformation, leveraging the Spark dataframe datastructure. The dataframe concept originates from R and then was adopted by Python Pandas with great success. The dataframe is the workhorse of the data scientist. The combination of Spark SQL and dataframe creates a powerful engine for data processing.

We are now gearing up for extracting the insights from the datasets using machine learning from Spark MLlib.