What is the Spark Streaming application data flow?

The following figure provides the data flow between the Spark driver, workers, streaming sources and targets:

What is the Spark Streaming application data flow?

It all starts with the Spark Streaming Context, represented by ssc.start() in the preceding figure:

  1. When the Spark Streaming Context starts, the driver will execute a long-running task on the executors (that is, the Spark workers).
  2. The Receiver on the executors (Executor 1 in this diagram) receives a data stream from the Streaming Sources. With the incoming data stream, the receiver divides the stream into blocks and keeps these blocks in memory.
  3. These blocks are also replicated to another executor to avoid data loss.
  4. The block ID information is transmitted to the Block Management Master on the driver.
  5. For every batch interval configured within Spark Streaming Context (commonly this is every 1 second), the driver will launch Spark tasks to process the blocks. Those blocks are then persisted to any number of target data stores, including cloud storage (for example, S3, WASB, and so on), relational data stores (for example, MySQL, PostgreSQL, and so on), and NoSQL stores.

Suffice it to say, there are a lot of moving parts for a streaming application that need to be continually optimized and configured. Most of the documentation for Spark Streaming is more complete in Scala, so, as you are working with the Python APIs, you may sometimes need to reference the Scala version of the documentation instead. If this happens to you, please file a bug and/or fill out a PR if you have a proposed fix (https://issues.apache.org/jira/browse/spark/).

For a deeper dive on this topic, please refer to:

  1. Spark 1.6 Streaming Programming Guide: https://spark.apache.org/docs/1.6.0/streaming-programming-guide.html
  2. Tathagata Das' Deep Dive with Spark Streaming (Spark Meetup 2013-06-17): http://www.slideshare.net/spark-project/deep-divewithsparkstreaming-tathagatadassparkmeetup20130617
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset