Welcome to Native Docker clustering with Swarm! This is a book about containers and distributed systems. We’ll show how to use the native Docker tools to model micro services, spawn tasks, scale up the size of your application, and push your containers to the Docker clustering limit! In a word, we’ll discuss Docker orchestration.
With the recent rise of Swarm Mode and the enablement of Swarm inside the Docker Engine itself, it turned out that the best way to orchestrate Docker is… Docker!
Good, but what do we mean by “orchestrate Docker”? What is orchestration? Better, what is an orchestra?
An orchestra is an ensemble of musicians led by a conductor, who dictates tempo, rhythm and shapes the sound. Strings, woodwinds, percussions, keyboards and other instruments follow the conductor’s direction to perform an astonishing symphony, for example Beethoven’s The Ninth.
Similarly, in a containers orchestration system the musicians are tasks, and the conductor is a leader service (Swarm primitives). Tasks don’t play a symphony, or at least not only: More abstractly they execute some computational job, for example they run a webserver. The conductor, Swarm, is responsible for their provisioning, their availability, their linkage, their scaling. This (and more) is what we mean by “Docker orchestration”.
This book shows how to provision such Docker “orchestras”, how to guarantee the availability of the service, how to connect tasks and how to scale the platform, to play the symphony of your application.
Chapter 1, Welcome to Docker Swarm, introduces Swarm, and explains why you need a clustering solution for your containers. It illustrates the Swarm features, giving a high-level description of its architecture. We define some use cases and describe how Swarm is different from Fleet, Kubernetes and Mesos. The chapter proceeds with the Docker tools installation and finally with two Swarms provisionings: A local Swarm Standalone and a remote Swarm Mode cluster on DigitalOcean.
Chapter 2, Discover the Discovery Services, is a descriptive and mostly abstract chapter. We’ll learn what discovery mechanisms and consensus algorithms are, and why they are crucial for distributed systems. We’ll describe in detail Raft and its implementation Etcd, the consensus mechanism included in Swarm Mode. We will also show the limitations of the discovery mechanism used in Chapter 1, Welcome to Docker Swarm, by extending the local tiny example with Consul, re deploying it.
Chapter 3, Meeting Docker Swarm Mode, is about the new Docker kit that allows to create task clusters of any size. We will introduce Swarmit, the foundation of Docker Swarm Mode, showing how it works in Docker 1.12+, discuss its architecture, its concepts, how it’s different from the “old” Swarm, and how it organizes workloads by abstracting services and tasks.
Chapter 4, Creating a Production-Grade Swarm, shows and discusses the community-driven projects Swarm2k and Swarm3k, our 2,300 and 4,800 nodes Swarm clusters experiments, which ran hundreds of thousands of containers. We demonstrate how such huge clusters were planned, provisioned, and summarize the lessons we learned.
Chapter 5, Administer a Swarm Cluster, is a chapter about infrastructure. We will show how to increase or decrease Swarms sizes, how to promote and demote nodes, and how to update clusters and nodes properties. We’ll introduce Shipyard and Portainer.io as graphical UIs for Swarm.
Chapter 6, Deploy Real Applications on Swarm, is where we will put real applications in motion on Swarm and where we add to the discussion some notes about Compose, Docker Stacks and Docker Application Bundles. We will show the typical deployment workflow, how to filter and schedule containers over the cluster, launch them as services, handle containers as tasks. We’ll start defining a web service with Nginx, then we’ll deploy a mandatory Wordpress with MySQL example. We’ll finally move on with a more realistic app: Apache Spark.
Chapter 7, Scale Up Your Platform, will develop new topics from the previous chapter. Here we’ll introduce Flocker to add storage capacity to Spark on Swarm, and we’ll show how to install and use it automatically at a scale in conjunction with Swarm. We’ll refine our Spark example by running some real big data jobs and setting up a basic monitoring system for this infrastructure.
Chapter 8, Exploring Additional Features to Swarm, discusses some advanced topics important to Swarm, such as Libnetwork and Libkv.
Chapter 9, Securing a Swarm Cluster and Docker Software Supply Chain, will focus on security considerations for Swarm clusters. Among the arguments, certificates, firewalling concepts for platform, and a mention to Notary.
Chapter 10, Swarm and the Cloud, is a chapter illustrating the most popular options for running Swarm on cloud providers. We’ll install Swarm on AWS and Azure before introducing the Docker Datacenter, and we’ll finally move to OpenStack showing how to install and manage Swarms on the top of Magnum, the Container as a Service solution for OpenStack.
Chapter 11, What is Next?, concludes the discussion with an overview on the next Docker orchestration trends, such as software defined infrastructures, Infrakit, unikernels, Containers as a Service. The adventure continues!