- Foreword
- Preface
- I. Fundamentals of Stream Processing with Apache Spark
- 1. Introducing Stream Processing
- 2. Stream-Processing Model
- 3. Streaming Architectures
- 4. Apache Spark as a Stream-Processing Engine
- 5. Spark’s Distributed Processing Model
- Running Apache Spark with a Cluster Manager
- Spark’s Own Cluster Manager
- Understanding Resilience and Fault Tolerance in a Distributed System
- Data Delivery Semantics
- Microbatching and One-Element-at-a-Time
- Bringing Microbatch and One-Record-at-a-Time Closer Together
- Dynamic Batch Interval
- Structured Streaming Processing Model
- 6. Spark’s Resilience Model
- A. References for Part I
- II. Structured Streaming
- 7. Introducing Structured Streaming
- 8. The Structured Streaming Programming Model
- 9. Structured Streaming in Action
- 10. Structured Streaming Sources
- 11. Structured Streaming Sinks
- 12. Event Time–Based Stream Processing
- 13. Advanced Stateful Operations
- 14. Monitoring Structured Streaming Applications
- 15. Experimental Areas: Continuous Processing and Machine Learning
- B. References for Part II
- III. Spark Streaming
- 16. Introducing Spark Streaming
- 17. The Spark Streaming Programming Model
- 18. The Spark Streaming Execution Model
- 19. Spark Streaming Sources
- 20. Spark Streaming Sinks
- 21. Time-Based Stream Processing
- 22. Arbitrary Stateful Streaming Computation
- 23. Working with Spark SQL
- 24. Checkpointing
- 25. Monitoring Spark Streaming
- 26. Performance Tuning
- C. References for Part III
- IV. Advanced Spark Streaming Techniques
- 27. Streaming Approximation and Sampling Algorithms
- Exactness, Real Time, and Big Data
- The Exactness, Real-Time, and Big Data triangle
- Approximation Algorithms
- Hashing and Sketching: An Introduction
- Counting Distinct Elements: HyperLogLog
- Counting Element Frequency: Count Min Sketches
- Ranks and Quantiles: T-Digest
- Reducing the Number of Elements: Sampling
- 28. Real-Time Machine Learning
- D. References for Part IV
- V. Beyond Apache Spark
- 29. Other Distributed Real-Time Stream Processing Systems
- 30. Looking Ahead
- E. References for Part V
- Index
Foreword
Welcome to Stream Processing with Apache Spark!
It’s very exciting to see how much both the Apache Spark project, as well as stream processing with Apache Spark have come along since it was first started by Matei Zaharia at University of California Berkeley in 2009. Apache Spark started off as the first unified engine for big data processing and has grown into the de-facto standard for all things big data.
Stream Processing with Apache Spark is an excellent introduction to the concepts, tools, and capabilities of Apache Spark as a stream processing engine. This book will first introduce you to the core Spark concepts necessary to understand modern distributed processing. Then it will explore different stream processing architectures and the fundamental architectural trade-offs between then. Finally, it will illustrate how Structured Streaming in Apache Spark makes it easy to implement distributed streaming applications. In addition, it will also cover the older Spark Streaming (aka, DStream) APIs for building streaming applications with legacy connectors.
In all, this book covers everything you’ll need to know to master building and operating streaming applications using Apache Spark! We look forward to hearing about what you’ll build!
Tathagata Das
Cocreator of Spark Streaming and Structured Streaming
Michael Armbrust
Cocreator of Spark SQL and Structured Streaming
Bill Chambers
Coauthor of Spark: The Definitive Guide
May 2019
-
No Comment