In this chapter, we will cover the following recipes:
In this chapter, we'll be looking at how to bundle our Spark application and deploy it on various distributed environments.
As we discussed earlier in Chapter 3, Loading and Preparing Data – DataFrame the foundation of Spark is the RDD. From a programmer's perspective, the composability of RDDs such as a regular Scala collection is a huge advantage. RDD wraps three vital (and two subsidiary) pieces of information that help in reconstruction of data. This enables fault tolerance. The other major advantage is that while the processing of RDDs could be composed into hugely complex graphs using RDD operations, the entire flow of data itself is not very difficult to reason with.
Other than optional optimization attributes, such as data location, an RDD at its core wraps only three vital pieces of information:
Spark spawns one task per partition. So, a partition is the basic unit of parallelism in Spark.
The number of partitions could be any of these:
spark.default.parallelism
parameter (set while starting the cluster)repartition
or coalesce
on the RDDSo far, we have just run our Spark application in the self-contained single JVM mode. While the programs work just fine, we have not yet exploited the distributed nature of the RDDs.
As always, all the code snippets for this chapter can be downloaded from https://github.com/arunma/ScalaDataAnalysisCookbook/tree/master/chapter6-scalingup.