- Big Data Analytics
- Table of Contents
- Big Data Analytics
- Credits
- About the Author
- Acknowledgement
- About the Reviewers
- www.PacktPub.com
- Preface
- 1. Big Data Analytics at a 10,000-Foot View
- 2. Getting Started with Apache Hadoop and Apache Spark
- 3. Deep Dive into Apache Spark
- Starting Spark daemons
- Learning Spark core concepts
- Lifecycle of Spark program
- Spark applications
- Persistence and caching
- Spark resource managers – Standalone, YARN, and Mesos
- Summary
- 4. Big Data Analytics with Spark SQL, DataFrames, and Datasets
- History of Spark SQL
- Architecture of Spark SQL
- Introducing SQL, Datasources, DataFrame, and Dataset APIs
- Evolution of DataFrames and Datasets
- Why Datasets and DataFrames?
- When to use RDDs, Datasets, and DataFrames?
- Analytics with DataFrames
- Analytics with the Dataset API
- Data Sources API
- Spark SQL as a distributed SQL engine
- Hive on Spark
- Summary
- 5. Real-Time Analytics with Spark Streaming and Structured Streaming
- 6. Notebooks and Dataflows with Spark and Hadoop
- 7. Machine Learning with Spark and Hadoop
- 8. Building Recommendation Systems with Spark and Mahout
- 9. Graph Analytics with GraphX
- 10. Interactive Analytics with SparkR
- Index
Apache Hive supports three execution engines—MapReduce, Tez, and Spark. Though Hivemall does not support Spark natively, the Hivemall for Spark project (https://github.com/maropu/hivemall-spark) implements a wrapper for Spark. This wrapper enables you to use Hivemall UDFs in SparkContext, DataFrames, or Spark Streaming. It is really easy to get started with Hivemall for Spark. Follow this procedure to start a Scala shell, load UDFs, and execute SQLs:
- Download the
define-udfsscript:[cloudera@quickstart ~]$ wget https://raw.githubusercontent.com/maropu/hivemall-spark/master/scripts/ddl/define-udfs.sh --no-check-certificate - Start a Scala shell with the
packagesoption:[cloudera@quickstart ~]$ spark-1.6.0-bin-hadoop2.6/bin/spark-shell --master local[*] --packages maropu:hivemall-spark:0.0.6 - Create Hivemall functions as follows. Hivemall for Spark does not support Python yet:
scala> :load define-udfs.sh - Now you can execute examples from:
https://github.com/maropu/hivemall-spark/tree/master/tutorials
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.