So let's divide our cluster into two tiers. We say tiers, but they can be any name you want, we just like the term "tier" and it is commonly used. We assume that we have six nodes. We want our more powerful nodes numbered 1 and 2 to be placed in the tier called hot and the nodes numbered 3, 4, 5, and 6, which are smaller in terms of CPU and memory, but very large in terms of disk space, to be placed in a tier called cold.

To configure, we add the following property to the elasticsearch.yml configuration file on nodes 1 and 2 (the ones that are more powerful):

node.tier: hot

Of course, we will add a similar property to the elasticsearch.yml configuration file on nodes 3, 4, 5, and 6 (the less powerful ones):

node.tier: cold

Now let's create our daily index for today's data, one called logs_2015-12-10. As we said earlier, we want this to be placed on the nodes in the hot tier. We do this by running the following commands:

curl -XPUT 'http://localhost:9200/logs_2015-12-10' -d '{
 "settings" : {
  "index" : {
   "routing.allocation.include.tier" : "hot"
  }
 }
}'

The preceding command will result in the creation of the logs_2015-12-10 index and specification of the index.routing.allocation.include.tier property to it. We set this property to the hot value, which means that we want to place the logs_2015-12-10 index on the nodes that have the node.tier property set to hot.

Now, when the day ends and we need to create a new index, we again put it on the hot nodes. We do this by running the following command:

curl -XPUT 'http://localhost:9200/logs_2015-12-11' -d '{
 "settings" : {
  "index" : {
   "routing.allocation.include.tier" : "hot"
  }
 }
}'

Finally, we need to tell Elasticsearch to move the index holding the data for the previous day to the cold tier. We do this by updating the index settings and setting the index.routing.allocation.include.tier property to cold. This is done using the following command:

curl -XPUT 'http://localhost:9200/logs_2015-12-10/_settings' -d '{
 "index.routing.allocation.include.tier" : "cold"
}'

After running the preceding command, Elasticsearch will start relocating the index called logs_2015-12-10 to the nodes that have the node.tier property set to cold in the elasticsearch.yml file without any manual work needed from us.

In the same manner as we specified on which nodes the index should be placed, we can also exclude nodes from index allocation. Referring to the previously shown example. if we want the index called logs_2015-12-10 to not be placed on the nodes with the node.tier property set to cold, we would run the following command:

curl -XPUT 'localhost:9200/logs_2015-12-10/_settings' -d '{
 "index.routing.allocation.exclude.tier" : "cold"
}'

Notice that instead of the index.routing.allocation.include.tier property, we've used the index.routing.allocation.exclude.tier property.

In addition to inclusion and exclusion rules, we can also specify the rules that must match in order for a shard to be allocated to a given node. The difference is that when using the index.routing.allocation.include property, the index will be placed on any node that matches at least one of the provided property values. Using index.routing.allocation.require, Elasticsearch will place the index on a node that has all the defined values. For example, let's assume that we've set the following settings for the logs_2015-12-10 index:

curl -XPUT 'localhost:9200/logs_2015-12-10/_settings' -d '{
 "index.routing.allocation.require.tier" : "hot",
 "index.routing.allocation.require.disk_type" : "ssd"
}'

After running the preceding command, Elasticsearch would only place the shards of the logs_2015-12-10 index on a node with the node.tier property set to hot and the node.disk_type property set to ssd.

Instead of adding a special parameter to the nodes configuration, we are allowed to use IP addresses to specify which nodes we want to include or exclude from the shards and replicas allocation. In order to do this, instead of using the tier part of the index.routing.allocation.include.tier or index.routing.allocation.exclude.tier properties, we should use the _ip. For example, if we would like our logs_2015-12-10 index to be placed only on the nodes with the 10.1.2.10 and 10.1.2.11 IP addresses, we would run the following command:

curl -XPUT 'localhost:9200/logs_2015-12-10/_settings' -d '{
 "index.routing.allocation.include._ip" : "10.1.2.10,10.1.2.11"
}'

In addition to the already described allocation filtering methods, Elasticsearch gives us disk-based shard allocation rules. It allows us to set allocation rules based on the nodes' disk usage.

curl -XPUT localhost:9200/_cluster/settings -d '{
 "transient" : {
  "cluster.routing.allocation.disk.threshold_enabled" : false
 }
}'

Allocation awareness allows us to configure shards and their replicas allocation with the use of generic parameters. In order to illustrate how allocation awareness works, we will use our example cluster. For the example to work, we should add the following property to the elasticsearch.yml file:

cluster.routing.allocation.awareness.attributes: group

This will tell Elasticsearch to use the node.group property as the awareness parameter.

After this, let's start the first two nodes, the ones with the node.group parameter equal to groupA, and let's create an index by running the following command:

curl -XPOST 'localhost:9200/awarness' -d '{
 "settings" : {
  "index" : {
   "number_of_shards" : 1,

   "number_of_replicas" : 1
  }
 }
}'

After this command, our two-node cluster will look more or less like this:

As you can see, the index was divided between the two nodes evenly. Now let's see what happens when we launch the rest of the nodes (the ones with node.group set to groupB):

Notice the difference—the primary shards were not moved from their original allocation nodes, but the replica shards were moved to the nodes with a different node.group value. That's exactly right; when using shard allocation awareness, Elasticsearch won't allocate the primary shards and replicas of the same index to the nodes with the same value of the property used to determine the allocation awareness (which in our case is the node.group).

Forcing allocation awareness can come in handy when we know, in advance, how many values our awareness attributes can take and we don't want more replicas than needed to be allocated in our cluster, for example, not to overload our cluster with too many replicas. For this, we can force the allocation awareness to be active only for certain attributes. We can specify these values using the cluster.routing.allocation.awareness.force.zone.values property and providing a list of comma-separated values to it. For example, if we would like the allocation awareness to use only the groupA and groupB values of the node.group property, we would add the following to the elasticsearch.yml file:

cluster.routing.allocation.awareness.attributes: group
cluster.routing.allocation.awareness.force.zone.values: groupA, groupB

Elasticsearch allows us to configure allocation for the entire cluster or for the index level. In the case of cluster allocation, we can use the properties prefixes:

When it comes to index-specific allocation, we can use the following properties prefixes:

The previously mentioned prefixes can be used with the properties that we've defined in the elasticsearch.yml file (our tag and group properties) and with a special property called _ip that allows us to match or exclude the use of the nodes' IP addresses, for example, like this:

cluster.routing.allocation.include._ip: 192.168.2.1

If we would like to include nodes with a group property matching the groupA value, we would set the following property:

cluster.routing.allocation.include.group: groupA

Notice that we've used the cluster.routing.allocation.include prefix and we've concatenated it with the name of the property, which is group in our case.

Let's say we have two nodes called es_node_one and es_node_two, and we have two shards of the shop index placed by Elasticsearch on the first node and we would like to move the second shard to the second node. In order to do this, we can run the following command:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [ { 
  "move" : {
   "index" : "shop", 
   "shard" : 1, 
   "from_node" : "es_node_one", 
   "to_node" : "es_node_two" 
  }
 } ]
}'

We've specified the move command, which allows us to move shards (and replicas) of the index specified by the index property. The shard property is the number of shards we want to move. And, finally, the from_node property specifies the name of the node we want to move the shard from and the to_node property specifies the name of the node we want the shard to be placed on.

If we would like to cancel an on-going allocation process, we can run the cancel command and specify the index, node, and shard we want to cancel the allocation for. For example:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [ {
  "cancel" : {
   "index" : "shop", 
   "shard" : 0, 
   "node" : "es_node_one"
  }
 } ]
}'

The preceding command would cancel the allocation of shard 0 of the shop index on the es_node_one node.

In addition to cancelling and moving shards and replicas, we are also allowed to allocate an unallocated shard to a specific node. For example, if we have an unallocated shard numbered 0 for the users index and we would like it to be allocated to es_node_two by Elasticsearch, we would run the following command:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [ {
  "allocate" : {
   "index" : "users", 
   "shard" : 0, 
   "node" : "es_node_two"
  }
 } ]
}'

We can, of course, include multiple commands in a single HTTP request. For example:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
 "commands" : [
  {"move" : {"index" : "shop", "shard" : 1, "from_node" : "es_node_one", "to_node" : "es_node_two"}},
  {"cancel" : {"index" : "shop", "shard" : 0, "node" : "es_node_one"}}
 ]
}'

The cancel and allocate commands accept an additional allow_primary parameter. If set to true, it tells Elasticsearch that the operation can be performed on the primary shard. Please be advised that operations with the allow_primary parameter set to true may result in data loss.