Service rebalancing has been one of the requested features for Docker. However, it is better to have this feature running outside the orchestrator and to run it as an operator container.

The problem is that after a new node joins the cluster, we usually rebalance the running services to spread loads across the cluster. The main reason this feature is not built into the orchestrator is because it is application-specific. Also, if the cluster keeps rebalancing everything when nodes dynamically come and go, running services may be broken all the time, and not in a good enough condition to serve requests.

However, if we implement this kind of feature as an operator container, we can optionally disable it when necessary as it is running outside the orchestrator. Also, we can selectively pick only particular services to be rebalanced.

The service balancer is currently available as chanwit/service-balancer on Docker's hub. We will be running only one instance of service balancer on any manager:

$ docker service create 
 --name service-balancer 
 --network control 
 --constraint node.role==manager 
 --replicas 1 
 chanwit/service-balancer

Something to consider when using the auto-rebalancer is that --update-delay must be set to greater than the startup time of each task. This is really important, especially for Java-based services. This delay should be large enough, at least larger than the interval used by the health checking mechanism.

Also, for the safest result, the value of --update-parallelism should start at 1, and gradually increase when the system can stably serve the requests.

To allow a service to automatically rebalance, the service balancer operator checks the service's label rebalance.on.node.create=true. If this label is present on the service, it will be rebalanced every time a new node is added to the cluster.