Elasticsearch is highly scalable and distributed in nature. This scalability would not have been possible without a reliable centralized co-ordination system. In fact, every distributed system requires a coordination system to maintain configuration information' and provide distributed synchronizations to all the nodes connected in the cluster. If you have worked with SolrCloud, you will know that it uses a coordination service known as Zookeeper. Zookeeper is an awesome open source project as a whole and can be used with many distributed systems, even with Elasticsearch by installing the plugin.
At the heart of Elasticsearch, there is no third-party coordination service. However, Elasticsearch does have a built-in centralized coordination mechanism—Zen Discovery. One of the primary tasks of Discovery is to choose a master node that looks after the entire cluster.
When you start a node in an Elasticsearch cluster, the first thing that happens is that a node searches for the master node that has the same cluster name. If it finds one, it simply joins the cluster, and if there is no master node available and this newly started node is master eligible (we will see what it means), then it becomes the master node. This process of cluster formation is known as discovery.
Zen-Discovery allows you to have two types of discovery mechanisms: Multicasting and Unicasting.
In the versions before 2.0, mulitcast used to be the default discovery type in Elasticsearch. However, since version 2.0, this has been changed to unicast discovery.
The reason was that multicasting is good to go in a development environment, but comes with added disadvantages. In multicasting, every node in the cluster sends message packets to all the other nodes for communication and health checks. This not only makes the network congested because of too many message transmissions in the network, it also makes it less secure. Any node that is unwanted and has the same cluster name can automatically join the cluster.
To enable the multicasting discovery type, add the following parameter in all the nodes of your cluster and restart them:
discovery.zen.ping.multicast.enabled: true
In unicasting discovery, the transmission of a single message is sent over the network to a single host at once. Here, you configure a set of nodes that can receive the messages from the node that wants to join the cluster. The unicasting mechanism is secure too since a node that wants to join the cluster must know the address and port of the master nodes that are responsible for deciding who will join the cluster.
To configure unicasting discovery, there are four properties that need to be configured inside the elasticsearch.yml file.
A split-brain is the situation in which one Elasticsearch cluster divides itself into two clusters, and each cluster has a separate master node. This is mainly caused by network issues or when a cluster becomes unstable because nodes experience long pauses due to slow garbage collections. The subsets of nodes attempt to form their own clusters, and this is known as a Split-Brain situation. In this situation, the nodes are diverged and cannot form a single cluster again until you kill the other half of the cluster. Split-brain can be very dangerous and can incur potential data loss. Luckily, the Elasticsearch team has worked hard to prevent the worst scenarios of handling split-brain situations, but the implementation is up to you.
To avoid split-brain, you need to decide the minimum number of master nodes that must be operational in the cluster to keep the cluster running. The size of the minimum number of master nodes depends on the total number of master nodes you have in your cluster. It can be set using the discovery.zen.minimum_master_nodes property inside the elasticsearch.yml file:
discovery.zen.minimum_master_nodes: n
Here, n is the integer value of the minimum number of master nodes. As per recommendation, this value should be decided based on the formula, N/2+1, where N is the total number of master nodes in the cluster. So, if you have three master nodes, this parameter can be set as 3/2+1 = 2 (rounding off to the nearest integer).
Unicasting requires an initial list of hosts to be pinged when a new node is started to form a cluster. Here you need to provide the list of all your master nodes along with an optional TCP port number in the following format:
discovery.zen.ping.unicast.hosts: ["masternode1IPAddress:TCP-Port","masternode2IPAddress:TCP-Port", "masternode3IPAddress:TCP-Port"]
The TCP port defaults to 9300.
This, by default, has 3s; within this time the nodes will ping to the master node and the master node will ping back to the other nodes to ensure that all nodes are up. This property should be set to a higher value in a slow network or a congested cluster. It can be configured in the following way:
discovery.zen.ping.timeout: 5s