Recall that we used the term Docker host to refer to a machine with Docker installed. When we join these hosts together to form a cluster, sometimes we call each of them a Docker node.

A Swarm cluster consists of two kinds of Docker nodes, a master and a worker. We say node mg0 has the master role, and node w01 has the worker role, for example. We form a cluster by joining other nodes to a master, usually the first master. The docker swarm join command requires the security tokens to be different, to allow a node to join as the master or as the worker. Please note that we must run the docker swarm join command on each node, not on the master node:

# Login to each node
$ docker swarm join --token SWMTKN-1-27uhz2azpesmsxu0tlli2e2uhdr2hudn3e2x5afilc02x1zicc-9wd3glqr5i92xmxvpnzdwz2j9 192.168.1.4:2377

A master node is responsible for controlling the cluster. The best practice recommended by Docker is that odd numbers of master nodes are the best configurations. We should have an odd number of master nodes starting from three. If we have three masters, one of them is allowed to fail and the cluster will still work.

The following table shows the possible configurations, from one to six master nodes. For example, a cluster of three master nodes allows one master to fail and it still maintains the cluster. If two masters fail, the cluster will not be allowed to operate, starting or stopping services. However, in that state, the running containers will not die and continue to run:

The best option to recover the cluster after losing the majority of master nodes is to bring the failed master nodes back online as fast as possible.

In the production cluster, we usually do not schedule running tasks on master nodes. A master node needs to have enough CPU, memory, and network bandwidth to properly handle node information and Raft logs. We control the cluster by commanding one of the master nodes. For example, we can list all nodes of a cluster by sending the following command to a master:

$ docker node ls

ID            HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
wbb8rb0xob *  mg0       Ready   Active        Leader

What we see in the result is the list of all nodes in the current cluster. We can tell that the mg0 node is a manager by looking at the MANAGER STATUS column. If a manager node is the primary manager of the cluster, MANAGER STATUS will say it is a Leader. If we have two more manager nodes here, the status will tell us they are a Follower. Here's how this leader/follower mechanism works. When we issue a command to the leader, the leader performs the command and the state of the cluster is changed. The cluster state is then updated by also sending this change to other manager nodes, that is, followers. If we issue a command to a follower, it will forward the command to the leader instead of doing that itself. Basically, all commands for the cluster will be performed by the leader, and the followers will update the changes to their internal Raft logs only.

If a new manager node would like to join, we require a master token for it. Type the docker swarm join-token manager command to obtain a security token to join a cluster in a manager role: 

$ docker swarm join-token manager

To add a manager to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-2c6finlm9d97q075kpwxcn59q93vbpfaf5qp13awjin3s3jopw-5hex62dfsd3360zxds46i6s56 192.168.1.4:2377

Although a task as a container can be running on both kinds of nodes, we usually do not submit tasks to run on master nodes. We only use worker nodes to run tasks in production. To join worker nodes to the cluster, we pass the worker token to the join command. Use docker swarm join-token worker to obtain a worker token.