Developing a Spark applications in Eclipse with Maven

Maven as a build tool has become the de-facto standard over the years. It's not surprising if we look a little deeper into what Maven brings. Maven has two primary features and they are:

  • Convention over configuration: Tools built prior to Maven gave developers the freedom to choose where to put source files, test files, compiled files, and so on. Maven takes away this freedom. Because of this, all the confusion about locations also disappears. In Maven, there is a specific directory structure for everything. The following table shows a few of the most common locations:
/src/main/scala Source code in Scala
/src/main/java Source code in Java
/src/main/resources Resources to be used by the source code, such as configuration files
/src/test/scala Test code in Scala
/src/test/java Test code in Java
/src/test/resources Resources to be used by the test code, such as configuration files
  • Declarative dependency management: In Maven, every library is defined by the following three coordinates:
groupId A logical way of grouping libraries similar to a package in Java/Scala, which has to be at least the domain name you own, for example, org.apache.spark.
artifactId The name of the project and JAR
version Standard version numbers

In pom.xml file (the configuration file that provides Maven all the information about a project), dependencies are declared in the form of these three coordinates. There is no need to search the internet and download, unpack, and copy libraries. All you need to do is provide three coordinates of the dependency JAR you need and Maven will do the rest for you. The following is an example of using a spark-core dependency:

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.0</version>
</dependency>

This makes dependency management, including transitive dependencies, very easy. Build tools that came after Maven, such as simple build tool (SBT) and Gradle, also follow these two rules as is and provide enhancements in other aspects.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset