Chapter 6 Introducing Spark and Kafka
by Sourabh Mukherjee, Amit Kumar Das, Sayan Goswami
Big Data Simplified
- Cover
- About Pearson
- Tittle
- Copyright
- Dedication
- Brief Contents
- Contents (1/2)
- Contents (2/2)
- Preface
- Acknowledgements
- About the Authors
- Model Syllabus for Big Data
- Lesson Plan
- Chapter 1 A Closer Look at Data
- Chapter 2 Introducing Big Data
- 2.1 Introduction
- 2.2 The Transition to Big Data
- 2.3 The Definition of Big Data
- 2.4 The V’s
- 2.5 Sources of Big Data
- 2.6 Common Applications of Big Data
- 2.7 An Introduction to Big Data Technologies
- 2.8 An Overview of Popular Vendors
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 3 Introducing Hadoop
- 3.1 Introduction
- 3.2 An Overview of Hadoop
- 3.3 Configuring a Hadoop Cluster (1/2)
- 3.3 Configuring a Hadoop Cluster (2/2)
- 3.4 Storing Data with HDFS
- 3.5 HDFS Technical Commands
- 3.6 Hadoop Distributions
- 3.7 Hadoop in the Cloud
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 4 Introducing MapReduce
- 4.1 Introduction
- 4.2 Processing Data with MapReduce
- 4.3 Parallelism in Map and Reduce Phases
- 4.4 Optimize the Map Phase Using a Combiner
- 4.5 What is YARN?
- 4.6 Example Use Case on MapReduce: Development and Execution Step-by-step (1/2)
- 4.6 Example Use Case on MapReduce: Development and Execution Step-by-step (2/2)
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 5 Introducing NoSQL
- 5.1 Introduction
- 5.2 NoSQL Databases in the Light of CAP Theorem
- 5.3 NoSQL Product Categories
- 5.4 NoSQL Database: Cassandra
- 5.4.1 Characteristics of Cassandra
- 5.4.2 Cassandra Architecture
- 5.4.3 Components of Cassandra
- 5.4.4 Cassandra Write Operations at a Node Level
- 5.4.5 Cassandra Node Level Read Operation
- 5.4.6 KEYSPACE in Cassandra
- 5.4.7 Starting Cassandra Server and Cqlsh Query Editor
- 5.4.8 DataStax Distribution Package
- 5.5 NoSQL Databases in the Cloud
- 5.6 NoSQL – Do’s and Don’ts
- 5.7 Business Intelligence and NoSQL
- 5.8 Big Data and NoSQL
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 6 Introducing Spark and Kafka
- 6.1 Introducing Spark
- 6.2 Working with Kafka
- 6.2.1 What is Apache Kafka
- 6.2.2 Kafka Architecture
- 6.2.3 Need of Apache Kafka in Big Data
- 6.2.4 Kafka Use Cases
- 6.2.5 Why is Kafka so Fast?
- 6.2.6 Kafka Needs ZooKeeper
- 6.2.7 Different Components in Kafka
- 6.2.8 Difference between Apache Kafka and Apache Flume
- 6.2.9 Kafka Demonstration—How Messages are Passing from Publisher to Consumer through a Topic
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 7 Other BigData Tools and Technologies
- Chapter 8 Working with Big Data in R
- 8.1 Prerequisites
- 8.2 Exploratory Data Analysis
- 8.3 R Libraries for Dealing with Large Data Sets
- 8.4 Integrating Hadoop with R
- 8.5 Simple R Program with Hadoop
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 9 Working with Big Data in Python
- Chapter 10 Big Data Applied
- Chapter 11 Big Data Strategy
- 11.1 Introduction
- 11.2 Two Typical Big Data Use Cases
- 11.3 Data Warehouses vs. Data Lakes—What is Your Strategy?
- 11.4 Key Questions to Ask
- 11.5 Getting Ready for a Big Data Program
- 11.6 Making Technology Choices
- 11.7 Making Tooling Choices
- Summary
- Short-answer Type Questions (5 Marks Questions)
- Long-answer Type Questions (10 Marks Questions)
- Chapter 12 Case Study: Retail Near Real-time Analytics
- 12.1 Introduction to Retail Domain
- 12.2 Near Real-time Analytics: Problem Statement
- 12.3 NRT Analytics: Solution Approach
- 12.4 NRT Analytics: Details of Solution Implemented (1/3)
- 12.4 NRT Analytics: Details of Solution Implemented (2/3)
- 12.4 NRT Analytics: Details of Solution Implemented (3/3)
- Summary
- Multiple-choice Questions (1 Mark Questions)
- Short-answer Type Questions (5 Marks Questions)
- Appendix (1/2)
- Appendix (2/2)
- Index
OBJECTIVE
CHAPTER
6
Introducing Spark
andKafka
6.1 Introducing Spark
6.2 Working with Kafka
Now we have covered the core Big Data components,
such as Hadoop, MapReduce and NoSQL. It is the right
time to introduce another very important aspect of the
Hadoop ecosystem, i.e., Apache Spark. Spark is widely
used across organizations to process large data sets.
It is extremely popular for its great processing speed
and ability to integrate with diverse databases. Apache
Spark is accompanied by Apache Kafka, an open source
distributed streaming platform which is used to stream
data. Developed in Scala and Java by LinkedIn, it was
contributed to the Apache Software Foundation. It pro-
vides uni ed, high-throughput, low-latency platform
for handling real-time data feeds. We shall study the
functions of Apache Kafka in this chapter.
M06 Big Data Simplified XXXX 01.indd 117 5/17/2019 2:49:07 PM
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.