A note on versions. Docker Swarm mode, which we'll introduce in the upcoming sections, is compatible only with Docker 1.12+. With Swarmkit, instead, you can orchestrate even previous versions of Docker Engines, for example, 1.11 or 1.10.

Swarmkit is the orchestration mechanism released to handle clusters of services of any size.

In a Swarmkit cluster, nodes can be either managers (of the cluster) or workers (the workhorses of cluster, nodes that execute compute operations).

There should be an odd number of managers, preferably 3 or 5, so that if there will be no split brains (as explained in Chapter 2, Discover the Discovery Services), and a majority of managers will drive the cluster. A quorum is always required by the Raft consensus algorithm.

A Swarmkit cluster can host any arbitrary number of workers: 1, 10, 100, or 2,000.

On managers, services may be defined and load balanced. For example, a service may be "web". A "web" service will be physically made of several tasks, running on the cluster nodes, including the managers, for example, one task can be a single nginx Docker container.

In Swarmkit, an operator uses a Swarmctl binary to interact with the system remotely, invoking operations on the leader master. Masters, that run a binary called Swarmd, agree on a leader via Raft, keep the status of services and tasks, and schedule jobs on workers.

Workers run the Docker Engine, and take jobs running them as separate containers.

The Swarmkit architecture can be subject to a redraw, but the core components (masters and workers) are going to stay. Rather, new objects can possibly be added with plugins, for allocating resources such as networks and volumes.

How a manager chooses the best node for a task

The way that Swarmkit spawns tasks over the cluster is called scheduling. The scheduler is an algorithm that uses criteria such as filters to decide where to physically start a task.

The core binary to start a SwarmKit service is called swarmd, and that's the daemon to create both the master and join slaves.

It can bind itself either to a local UNIX socket and to a TCP socket, but in both cases, is manageable by the swarmctl utility by connecting to (another) dedicated UNIX local socket.

In the example that follows in the next section, we'll use swarmd to create a first manager listening on port 4242/tcp, and then again we'll use swarmd on the other worker nodes, to make them join the manager, and finally we'll use swarmctl to check some facts about our cluster.

These binaries are encapsulated into the fsoppelsa/swarmkit image that's available on the Docker Hub, and that we're going to use here to simplify the explanation and avoid Go code compilation.

This is the online help for swarmd. It's rather self-explanatory in its tunables, so we're not going to cover all options in detail. For our practical purposes, the most important options are --listen-remote-api, defining the address:port for swarmd to bind on, and --join-addr, used from other nodes to join the cluster.

swarmctl is the client part of SwarmKit. It's the tool to use for operating SwarmKit clusters, as it is capable of showing the list of joined nodes, the list of services and tasks, and other information. Here, again from fsoppelsa/swarmkit, the swarmctl online help:

In this section, we'll provision a SwarmKit cluster initially made of a single manager and an arbitrary number of slaves.

To create such a setup, we'll use Ansible to make operations repeatable and more robust and, besides illustrating the commands, we'll proceed by examining the playbooks structure. You can easily adapt those playbooks to run on your provider or locally, but here we'll go on Amazon EC2.

To make this example run, there are some basic requirements.

If you want to follow the example on AWS, of course you must have an AWS account and have the access keys configured. Keys are retrievable from the AWS Console under your Account Name | Security Credentials. You will need to copy the following key's values:

I use awsctl to set those keys. Just install it from brew (Mac) or from your packaging system if you're using Linux or Windows, and configure it:

aws configure

Answer the prompt questions by pasting the keys when required. Configuration, where you can specify, for example, a favorite AWS region (such as us-west-1) is stored in ~/.aws/config, while credentials are in ~/.aws/credentials. In this way, keys are configured and read automatically by Docker Machine.

If you want to run the Ansible example instead of the commands, these are the software requirements:

Moreover, the example is using the standard port 4242/tcp, to make the cluster nodes interact with each other. So it's required to open that port in a security group.

Clone the repository at https://github.com/fsoppelsa/ansible-swarmkit and begin by setting up the SwarmKit Manager node:

ansible-playbook aws_provision_master.yml

After some docker-machine setup, the playbook will start a container on the Manager host, acting as a SwarmKit Manager. Here is the play snippet:

- name: Run the Swarmkit Master 
  docker: 
  name: swarmkit-master 
  image: "fsoppelsa/swarmkit" 
  command: swarmd --listen-remote-api 0.0.0.0:4242 
  expose: 
    - "4242" 
  ports: 
    - "0.0.0.0:4242:4242/tcp" 
  volumes: 
    - "/var/run/docker.sock:/var/run/docker.sock" 
  detach: yes 
  docker_url: "{{ dhost }}" 
  use_tls: encrypt 
  tls_ca_cert: "{{ dcert }}/ca.pem" 
  tls_client_cert: "{{ dcert }}/cert.pem" 
  tls_client_key: "{{ dcert }}/key.pem" 

On host, a container named swarmkit-master from the image fsoppelsa/swarmkit runs swarmd in manager mode (it listens at 0.0.0.0:4242). The swarmd binary uses the Docker Engine on host directly, so Engine's socket is mounted inside container. The container maps port 4242 to the host port 4242, so that swarmd is reachable by slaves directly by connecting to the host 4242 port.

In practice, it's the equivalent of this Docker command:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 
    4242:4242 fsoppelsa/swarmkit swarmd --listen-remote-api  
    0.0.0.0:4242

This command runs in detached mode (-d), passes via volumes (-v) the Docker machine Docker socket inside the container, exposes port 4242 from container to host (-p), and runs swarmd by putting the container itself in listening mode on any address, on port 4242.

Once the playbook has finished, you can source the swarmkit-master machine credentials and check whether our container is running correctly:

Now it's time to join some slaves. To start a slave, you can, guess what, just run:

ansible-playbook aws_provision_slave.yml

But since we want to join at least several nodes to the SwarmKit cluster, we go with a little bit of shell scripting:

for i in $(seq 5); do ansible-playbook aws_provision_slave.yml; 
    done

This command runs five times the playbook, thus creating five worker nodes. The playbook, after creating a machine named swarmkit-RANDOM will start a fsoppelsa/swarmkit container doing the following:

- name: Join the slave to the Swarmkit cluster
  docker:
    name: "{{machine_uuid}}"
    image: "fsoppelsa/swarmkit"
    command: swarmd --join-addr "{{ masterip }}":4242
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    detach: yes
    docker_url: "{{ shost }}"

Here, swarmd runs in join mode, and joins the cluster initiated on the Master, by connecting to port 4242/tcp. This is the equivalent of the following docker command:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock 
    fsoppelsa/swarmkit swarmd --join-addr $(docker-machine ip swarmkit- 
    master):4242

The ansible loop command will take some minutes to finish, depending on how many workers are starting. When the playbook has finished, we can control that the cluster was created correctly using swarmctl. If you haven't sourced the swarmkit-master Machine credential yet, it's time to:

eval $(docker-machine env swarmkit-master)

Now we invoke the container running the swarmd master, using exec:

docker exec -ti 79d9be555dab swarmctl -s /swarmkitstate/swarmd.sock 
    node ls

So, here we have listed the workers that have joined the master.

Using the usual swarmctl binary, we can now create a service (web), made of nginx containers.

We begin by checking to make sure that there are no active services on this brand new cluster:

So we're ready to start one, with this command:

docker exec -ti 79d9be555dab swarmctl service create --name web --
    image nginx --replicas 5

This command specifies to create a service named web, made of nginx container images, and replicate it with a factor of 5, so as to create 5 nginx containers across the cluster. It will take some seconds to take effect, because on each node of the cluster, Swarm will pull and start the nginx image, but finally:

The 5/5 indicates that of 5 desired replicas, 5 are up. We can see in detail where those containers got spawned using swarmctl task ls:

But, wait, is a nginx service (web.5) running on the manager node? Yes. SwarmKit and Swarm Mode managers are allowed to run tasks, by default, and the scheduler can dispatch jobs onto them.

In a real production configuration, if you want to reserve managers to not run jobs, you need to apply a configuration with labels and constraints. This is a topic of Chapter 5, Administer a Swarm Cluster.