- Foreword
- Preface
- I. Fundamentals of Stream Processing with Apache Spark
- 1. Introducing Stream Processing
- 2. Stream-Processing Model
- 3. Streaming Architectures
- 4. Apache Spark as a Stream-Processing Engine
- 5. Spark’s Distributed Processing Model
- Running Apache Spark with a Cluster Manager
- Spark’s Own Cluster Manager
- Understanding Resilience and Fault Tolerance in a Distributed System
- Data Delivery Semantics
- Microbatching and One-Element-at-a-Time
- Bringing Microbatch and One-Record-at-a-Time Closer Together
- Dynamic Batch Interval
- Structured Streaming Processing Model
- 6. Spark’s Resilience Model
- A. References for Part I
- II. Structured Streaming
- 7. Introducing Structured Streaming
- 8. The Structured Streaming Programming Model
- 9. Structured Streaming in Action
- 10. Structured Streaming Sources
- 11. Structured Streaming Sinks
- 12. Event Time–Based Stream Processing
- 13. Advanced Stateful Operations
- 14. Monitoring Structured Streaming Applications
- 15. Experimental Areas: Continuous Processing and Machine Learning
- B. References for Part II
- III. Spark Streaming
- 16. Introducing Spark Streaming
- 17. The Spark Streaming Programming Model
- 18. The Spark Streaming Execution Model
- 19. Spark Streaming Sources
- 20. Spark Streaming Sinks
- 21. Time-Based Stream Processing
- 22. Arbitrary Stateful Streaming Computation
- 23. Working with Spark SQL
- 24. Checkpointing
- 25. Monitoring Spark Streaming
- 26. Performance Tuning
- C. References for Part III
- IV. Advanced Spark Streaming Techniques
- 27. Streaming Approximation and Sampling Algorithms
- Exactness, Real Time, and Big Data
- The Exactness, Real-Time, and Big Data triangle
- Approximation Algorithms
- Hashing and Sketching: An Introduction
- Counting Distinct Elements: HyperLogLog
- Counting Element Frequency: Count Min Sketches
- Ranks and Quantiles: T-Digest
- Reducing the Number of Elements: Sampling
- 28. Real-Time Machine Learning
- D. References for Part IV
- V. Beyond Apache Spark
- 29. Other Distributed Real-Time Stream Processing Systems
- 30. Looking Ahead
- E. References for Part V
- Index
Part II. Structured Streaming
In this part, we examine Structured Streaming.
We begin our journey by exploring a practical example that should help you build your intuition for the model. From there, we examine the API and get into the details of the following aspects of stream processing:
-
Consuming data using sources
-
Building data-processing logic using the rich Streaming Dataframe/Dataset API
-
Understanding and working with event time
-
Dealing with state in streaming applications
-
Learning about arbitrary stateful transformations
-
Writing the results to other systems using sinks
Before closing, we provide an overview of the operational aspects of Structured Streaming.
Finally, we explore the current developments in this exciting new streaming API and provide insights into experimental areas like machine learning applications and near-real-time data processing with continuous streaming.
-
No Comment