In this chapter, we're going to extend what we saw in Chapter 6, Deploy Real Applications on Swarm. Our goal is to deploy a realistic production-grade Spark cluster on top of Swarm, add storage capacity, launch some Spark jobs and setup monitoring for the underlying infrastructure.
In order to do that, this chapter is mostly infrastructure-oriented. In fact, we'll see how to coalesce Libnetwork, Flocker, and Prometheus with Swarm.
For network, we'll use the basic Docker Network overlay system, based on Libnetwork. There are a few great networking plugins out there, such as Weave and others, but either they are not compatible with the new Docker Swarm Mode yet, or they are made obsolete by Swarm-integrated routing mesh mechanisms.
For storage, the situation is more prosperous, because there is much more choice (refer to https://docs.docker.com/engine/extend/plugins/). We'll go with Flocker. Flocker is the grandfather of Docker storage, and can be configured with a vast plethora of storage backends, making it one of the best choices for production loads. Scared by Flocker complexity? Unjustified: We'll see how to set up a multiple nodes Flocker cluster for any usage, in minutes.
For monitoring, finally, we'll introduce Prometheus. It's the most promising among the monitoring systems available for Docker nowadays, and its APIs may be integrated into the Docker engine very soon.
So, what we'll cover here:
- A Spark example over Swarm, ready for running any Spark job
- Automate the installation of Flocker for infrastructures at a scale
- Demonstrate how to use Flocker locally
- Use Flocker with Swarm Mode
- Scale our Spark app
- Monitor the health of this infrastructure with Prometheus
We're going to rearchitect the example of Chapter 6, Deploy Real Applications on Swarm, so we'll deploy Spark on Swarm, but this time with a realistic networking and storage setup.
Spark storage backend usually runs on Hadoop, or on NFS when on filesystem. For jobs not requiring storage, Spark will create local data on workers, but for storage computations, you will need a shared filesystem on each node, which cannot be guaranteed automatically by Docker volume plugins (at least, so far).
A possibility to achieve that goal on Swarm is to create NFS shares on each Docker host, and then mount them transparently inside service containers.
Our focus here is not to illustrate Spark job details and their storage organization, but to introduce an opinionated storage option for Docker and give an idea of how to organize and scale a fairly-complex service on Docker Swarm.