At this point, we checked that the Swarm was operative, by creating a service nginx with 3 replicas:
$ eval $(docker-machine env mg0) $ docker service create --name nginx --replicas 3 nginx du2luca34cmy
Afteer that, we found where the net namespace ID of the running Nginx was. We connected to mg0 via SSH to mg0 via SSH. The network namespace for Swarm's routing mesh was the one having the same timestamp as the special network namespace, 1-5t4znibozx. In this example, the namespace we're looking for is fe3714ca42d0.
root@mg0:~# ls /var/run/docker/netns -al total 0 drwxr-xr-x 2 root root 120 Aug 22 15:38 . drwx------ 5 root root 100 Aug 22 13:39 .. -r--r--r-- 1 root root 0 Aug 22 15:17 1-5t4znibozx -r--r--r-- 1 root root 0 Aug 22 15:36 d9ef48834a31 -r--r--r-- 1 root root 0 Aug 22 15:17 fe3714ca42d0
We can figure out our IPVS entries with ipvsadm, and run it inside the net namespace using the nsenter tool (https://github.com/jpetazzo/nsenter), as follows:
root@node1:~# nsenter --net=/var/run/docker/netns/fe3714ca42d0 ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 259 rr -> 10.255.0.8:0 Masq 1 0 2
Here, we can notice that there is an active round-robin IPVS entry. IPVS is the kernel-level load balancer, and it's used by Swarm to balance traffic in conjunction with iptables, which is used to forward and filter packets.
After cleaning the nginx test service (docker service rm nginx), we will set the managers in Drain mode, so to avoid them to take tasks:
$ docker node update --availability drain mg0 $ docker node update --availability drain mg1 $ docker node update --availability drain mg2
Now, we are ready to announce the availability of our managers on Twitter and Github and start the experiment!
Our contributors started joining their nodes to manager mg0 as workers. Anyone used their own favorite method, including the following:
- Looping
docker-machine ssh sudo docker swarm joincommands - Ansible
- Custom scripts and programs
We'll cover some of these methods in Chapter 5, Administer a Swarm Cluster .
After some time, we reached the quota of 2,300 workers and launched an alpine service with a replica factor of 100,000:
After some time, we reached the maximum of capacity for our managers, and we had to increase their physical resources. Live upgrading and maintenance of managers may be an expected operation in production. Here is how we did this operation.
With an odd number for the quorum, it is safe to demote a manager for maintenance.
$ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 4viybni..h24zxde mg1 Ready Active Reachable 6xxwumb..j6zvtyg * mg0 Ready Active Leader f1vs2e3..abdehnh mg2 Ready Active
Here, we had mg1 as a reachable manager, and we demoted it to worker with the following syntax:
$ docker node demote mg1 Manager mg1 demoted in the swarm.
We can see that the Reachable status of mg1 disappears from node ls output when it becomes a worker.
$ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 4viybni..h24zxde mg1 Ready Active 6xxwumb..j6zvtyg * mg0 Ready Active Leader f1vs2e3..abdehnh mg2 Ready Active
When the node is not a manager anymore, it's safe to shut it down, for example, with the DigitalOcean CLI, as we did:
$ doctl compute droplet-action shutdown 23362382
Listing the nodes, we noticed that mg1 was already down.
$ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 4viybni0ud2gjpay6ih24zxde mg1 Down Active 6xxwumbdac34bbgh6hj6zvtyg * mg0 Ready Active Leader f1vs2e3hjiqjaukmjqabdehnh mg2 Ready Active
We upgraded its resources to have 16G of memory, and then we powered the machine on again:
$ doctl -c .doctlcfg compute droplet-action power-on 23362382
When listing this time, we can expect some delay as mg1 is being back and reentered the cluster.
$ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 4viybni..h24zxde mg1 Ready Active 6xxwumb..j6zvtyg * mg0 Ready Active Leader f1vs2e3..abdehnh mg2 Ready Active
Finally, we can promote it back to the manager, as follows:
$ docker node promote mg1 Node mg1 promoted to a manager in the swarm.
Once this is done, the cluster operated normally. So, we repeated the operation for mg0 and mg2.