We have already seen how to test your Scala code using built-in ScalaTest package of Scala. However, in this subsection, we will see how we could test our Spark application written in Scala. The following three methods will be discussed:
- Method 1: Testing Spark applications using JUnit
- Method 2: Testing Spark applications using ScalaTest package
- Method 3: Testing Spark applications using Spark testing base
Methods 1 and 2 will be discussed here with some practical codes. However, a detailed discussion on method 3 will be provided in the next subsection. To keep the understanding easy and simple, we will use the famous word counting applications to demonstrate methods 1 and 2.