There's more...

One addition that Structured Streaming has done under the hood is adding the incremental plan execution. To understand it better, let's revisit a diagram from the Understanding the Catalyst optimizer recipe from Chapter 3, Spark SQL:

So the planner is aware that when streaming the logical plan, it should be converted into incremental physical execution plans. 

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset