The first group of metrics aggregations that we want to show you is the one that calculates the basic value from the given documents. These aggregations are:

As you can see, the preceding mentioned aggregations are pretty self-explanatory. So, let's try to calculate the average value on our data. For example, let's assume that we want to calculate the average number of copies for our books. The query to do that will look as follows:

{
 "aggs" : {
  "avg_copies" : {
   "avg" : {
    "field" : "copies"
   }
  }
 }
}

The results returned by Elasticsearch after running the preceding query will be as follows:

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "avg_copies" : {
      "value" : 1.75
    }
  }
}

So, we have an average of 1.75 copies per book. It is very easy to calculate – (6 + 0 + 1 + 0) / 4 is equal to 1.75. Seems that we got it right.

{
 "aggs" : {
  "avg_copies" : {
   "avg" : {
    "field" : "copies",
    "missing" : 0
   }
  }
 }
}

{
 "aggs": {
  "min_year": {
   "min": {
    "script": "doc['year'].value - 1000"
   }
  }
 }
}

{
 "aggs": {
  "min_year": {
   "min": {
    "field" : "year",
    "script" : {
     "inline" : "_value - factor",
     "params" : {
      "factor" : 1000
     }
    }
   }
  }
 }
}

curl -XGET 'localhost:9200/library/_search?size=0&pretty' -d '{
 "aggs": {
  "min_year": {
   "min": {
    "script": "doc["year"].value - 1000"
   }
  }
 }
}'

The next aggregations we will discuss are the ones that provide us with the statistical information about the numeric field we are running the aggregation on: the stats and extended_stats aggregations.

For example, the following query provides extended statistics for the year field:

{
 "aggs" : {
  "extended_statistics" : {
   "extended_stats" : {
    "field" : "year"
   }
  }
 }
}

The response to the preceding query will be as follows:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "extended_statistics" : {
      "count" : 4,
      "min" : 1886.0,
      "max" : 1961.0,
      "avg" : 1928.0,
      "sum" : 7712.0,
      "sum_of_squares" : 1.4871654E7,
      "variance" : 729.5,
      "std_deviation" : 27.00925767213901,
      "std_deviation_bounds" : {
        "upper" : 1982.018515344278,
        "lower" : 1873.981484655722
      }
    }
  }
}

As you can see, in the response we got information about the number of documents with value in the year field, the minimum value, the maximum value, the average, and the sum. These are the values that we will get if we run the stats aggregation instead of extended_stats. The extended_stats aggregation provides additional information, such as the sum of squares, variance, and standard deviation. Elasticsearch provides two types of aggregations because extended_stats is slightly more expensive when it comes to processing power.

The value_count aggregation is a simple aggregation which allows counting values in aggregated documents. This is quite useful when used with nested aggregations. We are not focusing on that topic right now, but it is something to keep in mind. For example, to use the value_count aggregation on the copied field, we will run the following query:

{
 "aggs" : {
  "count" : {
   "value_count" : {
    "field" : "copies"
   }
  }
 }
}

One of the aggregation that allows us to control how resource hungry the aggregation will be by controlling its precision, the cardinality aggregation calculates the count of distinct values in a given field. However, one thing needs to be remembered: the calculated count is an approximation, not the exact value. Elasticsearch uses the HyperLogLog++ algorithm (http://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/40671.pdf) to calculate the value.

This aggregation has a wide variety of use cases, such as showing the number of distinct values in a field that is responsible for holding the status code for your indexed Apache access logs. One query, and you know the approximated count of the distinct values in that field.

For example, we can request the cardinality for our title field:

{
 "aggs" : {
  "card_title" : {
   "cardinality" : {
    "field" : "title"
   }
  }
 }
}

To control the precision of the cardinality calculation, we can specify the precision_threshold property – the higher the value, the more precise the aggregation will be and the more resources it will need. The current maximum precision_threshold value is 40000 and the default depends on the parent aggregation. An example query using the precision_threshold property looks as follows:

{
 "aggs" : {
  "card_title" : {
   "cardinality" : {
    "field" : "title",
    "precision_threshold" : 1000
   }
  }
 }
}

The percentiles aggregation is another example of aggregation in Elasticsearch. It uses an algorithmic approximation approach to provide us with results. It uses the T-Digest algorithm (https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf) from Ted Dunning and Otmar Ertl and allows us to calculate percentiles: metrics that show us how many results are above a certain value. For example, the 99th percentile shows us the value that is greater than 99 percent of the other values.

Let's go into an example and look at a query that will calculate percentiles for the year field in our data:

{
 "aggs" : {
  "copies_percentiles" : {
   "percentiles" : {
    "field" : "year"
   }
  }
 }
}

The results returned by Elasticsearch for the preceding request will look as follows:

{
  "took" : 26,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "copies_percentiles" : {
      "values" : {
        "1.0" : 1887.2899999999997,
        "5.0" : 1892.4499999999998,
        "25.0" : 1918.25,
        "50.0" : 1932.5,
        "75.0" : 1942.25,
        "95.0" : 1957.25,
        "99.0" : 1960.25
      }
    }
  }
}

As you can see, the value that is higher than 99 percent of the values is 1960.25.

You may wonder why such aggregation is important. It is very useful for performance metrics; for example, where we usually look at averages for some period of time. Imagine that the average response time of our queries for the last hour is 50 milliseconds, which is not bad. However, if the 95th percentile would show 2 seconds, that would mean that about 5 percent of the users had to wait two or more seconds for the search results, which is not that good.

By default, the percentiles aggregation calculates seven percentiles: 1, 5, 25, 50, 75, 95, and 99. We can control this by using the percents property and specify which percentiles we are interested in. For example, if we want to get only the 95th and the 99th percentile, we change our query to the following one:

{
 "aggs" : {
  "copies_percentiles" : {
   "percentiles" : {
    "field" : "year",
    "percents" : [ "95", "99" ]
   }
  }
 }
}

We've mentioned earlier that the percentiles aggregation uses an algorithmic approach and is an approximation. As with all approximations, we can control the precision and memory usage of the algorithm. We do that by using the compression property, which defaults to 100. It is an internal property of Elasticsearch and its implementation details may change between versions. It is worth knowing that setting the compression value to one higher than 100 can increase the algorithm precision at the cost of memory usage.

The percentile_ranks aggregation is similar to the percentiles one that we just discussed. It allows us to show which percentile a given value has. For example, to show us which percentile year 1932 and year 1960 are, we run the following query:

{
 "aggs" : {
  "copies_percentile_ranks" : {
   "percentile_ranks" : {
    "field" : "year",
    "values" : [ "1932", "1960" ]
   }
  }
 }
}

The response returned by Elasticsearch will be as follows:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "copies_percentile_ranks" : {
      "values" : {
        "1932.0" : 49.5,
        "1960.0" : 61.5
      }
    }
  }
}

The top_hits aggregation keeps track of the most relevant document being aggregated. This doesn't sound very appealing, but it allows us to implement one of the most desired functionalities in Elasticsearch called document grouping, field collapsing, or document folding. Such functionality is very useful in some use cases—for example, when we want to show a book catalog but only one from a single publisher. To do that without the top_hits aggregation, we would have to run multiple queries. With the top_hits aggregation, we need only a single query.

The top_hits aggregation was introduced in Elasticsearch 1.3. In fact, the mentioned document folding is more or less a side effect and only one of the possible usage examples of the top_hits aggregation.

The idea behind the top_hits aggregation is simple. Every document that is assigned to a particular bucket can be also remembered. By default, only three documents per bucket are remembered.

To show you a potential use case that leverages the top_hits aggregation, we have decided to use a simple example. We would like to get the most relevant book published every 100 years. To do that we use the following query:

{
 "aggs": {
  "when": {
   "histogram": {
    "field": "year",
    "interval": 100
   },
   "aggs": {
    "book": {
     "top_hits": {
      "_source": {
       "include": [ "title", "available" ]
      },
      "size": 1
     }
    }
   }
  }
 }
}

In the preceding example, we did the histogram aggregation on year ranges. Every bucket was created for every one hundred years. The nested top_hits aggregations remembers a single document with the greatest score from each bucket (because of the size property being set to 1). We added the include option only for simpler results, so that we only return the title and available fields for every aggregated document. The response returned by Elasticsearch will be as follows:

{
  "took" : 8,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "when" : {
      "buckets" : [ {
        "key" : 1800,
        "doc_count" : 1,
        "book" : {
          "hits" : {
            "total" : 1,
            "max_score" : 1.0,
            "hits" : [ {
              "_index" : "library",
              "_type" : "book",
              "_id" : "4",
              "_score" : 1.0,
              "_source" : {
                "available" : true,
                "title" : "Crime and Punishment"
              }
            } ]
          }
        }
      }, {
        "key" : 1900,
        "doc_count" : 3,
        "book" : {
          "hits" : {
            "total" : 3,
            "max_score" : 1.0,
            "hits" : [ {
              "_index" : "library",
              "_type" : "book",
              "_id" : "2",
              "_score" : 1.0,
              "_source" : {
                "available" : false,
                "title" : "Catch-22"
              }
            } ]
          }
        }
      } ]
    }
  }
}

We can see that, because of the top_hits aggregation, we have the most scoring document (from each bucket) included in the response. In our particular case, the query was the match_all one and all the documents had the same score, so the top-scoring document for every bucket was more or less random. However, you need to remember that this is the default behavior. If we want to have custom sorting, this is not a problem for Elasticsearch. We just need to add the sort property for our top_hits aggregator. For example, we can return the first book from a given century:

{
 "aggs": {
  "when": {
   "histogram": {
    "field": "year",
    "interval": 100
   },
   "aggs": {
    "book": {
     "top_hits": {
      "sort": {
       "year": "asc"
      },
      "_source": {
       "include": [ "title", "available" ]
      },
      "size": 1
     }
    }
   }
  }
 }
}

We added sorting to the top_hits aggregation, so the results are sorted on the basis of the year field. This means that the first document will be the one with the lowest value in that field and this is the document that is going to be returned for each bucket.

{
 "aggs": {
  "when": {
   "histogram": {
    "field": "year",
    "interval": 100
   },
   "aggs": {
    "book": {
     "top_hits": {
      "highlight": {
       "fields": {
       "title": {}
       }
      },
      "explain": true,
      "version": true,
      "_source": {
       "include": [ "title", "available" ]
      },
      "fielddata_fields" : ["title"],
      "script_fields": {
       "century": {
        "script": "(doc["year"].value / 100).intValue()"
       }
      },
      "size": 1
     }
    }
   }
  }
 }
}

The geo_bounds aggregation is a simple aggregation that allows us to compute the bounding box that includes all the geo_point type field values from the aggregated documents.

We only need to provide the field (by using the field property; it needs to be of the geo_point type). We can also provide wrap_longitude (values true or false; it defaults to true) if the bounding box is allowed to overlap the international date line. In response, we get the latitude and longitude of the top-left and bottom-right corners of the bounding box. An example query using this aggregation looks as follows (using the hypothetical location field):

{
 "aggs" : {
  "box" : {
   "geo_bounds" : {
    "field" : "location"
   }
  }
 }
}

The last metric aggregation we want to discuss is the scripted_metric aggregation, which allows us to define our own aggregation calculation using scripts. For this aggregation, we can provide the following scripts (map_script is the only required one, the rest are optional):

For example, we can use the scripted_metric aggregation to calculate all the copies of all the books we have in our library by running the following request (we show the whole request to show how the names are escaped):

curl -XGET 'localhost:9200/library/_search?size=0&pretty' -d '{
 "aggs" : {
  "all_copies" : {
   "scripted_metric" : {
    "init_script" : "_agg["all_copies"] = 0",
    "map_script" : "_agg.all_copies += doc.copies.value",
    "combine_script" : "return _agg.all_copies",
    "reduce_script" : "sum = 0; for (number in _aggs) { sum += number }; return sum"
   }
  }
 }
}'

Of course, the preceding script is just a simple sum and we could use sum aggregation, but we just wanted to show you a simple example of what you can do with the scripted_metric aggregation.

As you can see, the init_script part of the aggregation is used to initialize the all_copies variable. Next, we have map_script, which is executed once for every document and we just add the value of the copies field to the earlier initialized variable. The combine_script part, executed once on each shard, tells Elasticsearch to return the calculated variable. Finally, the reduce_script part, executed once for the whole query on the aggregator node, will run a for loop, which will go through all the returned values that are stored in the _aggs array and return the sum of those. The final result returned by Elasticsearch for the preceding query looks as follows:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "all_copies" : {
      "value" : 7
    }
  }
}

The filter aggregation is a simple bucketing aggregation that allows us to filter the results to a single bucket. For example, let's assume that we want to get a count and the average copies count of all the books that are novels, which means they have the term novel in the tags field. The query that will return such results looks as follows:

{
 "aggs" : {
  "novels_count" : {
   "filter" : {
    "term": {
     "tags": "novel"
    }
   },
   "aggs" : {
    "avg_copies" : {
     "avg" : {
      "field" : "copies"
     }
    }
   }
  }
 }
}

As you can see, we defined the filter in the filter section of the aggregation definition and we defined a second nested aggregation. The nested aggregation is the one that will be run on the filtered documents.

The response returned by Elasticsearch looks as follows:

{
  "took" : 13,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "novels_count" : {
      "doc_count" : 2,
      "avg_copies" : {
        "value" : 3.5
      }
    }
  }
}

In the returned bucket, we have information about the number of documents (represented by the doc_count property) and the average number of copies, which is all we wanted.

The second bucket aggregation we want to show you is the filters aggregation. While the previously discussed filter aggregation resulted in a single bucket, the filters aggregation returns multiple buckets – one for each of the defined filters. Let's extend our previous example and assume that, in addition to the average number of copies for the novels, we also want to know the average number of copies for the books that are available. The query that will get us this information will use the filters aggregation and will look as follows:

{
 "aggs" : {
  "count" : {
   "filters" : {
    "filters" : {
     "novels" : {
      "term" : {
       "tags" : "novel"
      }
     },
     "available" : {
      "term" : {
       "available" : true
      }
     }
    }
   },
   "aggs" : {
    "avg_copies" : {
     "avg" : {
      "field" : "copies"
     }
    }
   }
  }
 }
}

Let's stop here and look at the definition of the aggregation. As you can see, we defined two filters using the filters section of the filters aggregation. Each filter has a name and the actual Elasticsearch filter; the first is called novels and the second is called available. Elasticsearch will use these names in the returned response. The thing to remember is that Elasticsearch will create a bucket for each defined filter and will calculate the nested aggregation that we defined – in our case, the one that calculates the average number of copies.

The results returned by Elasticsearch are as follows:

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "count" : {
      "buckets" : {
        "novels" : {
          "doc_count" : 2,
          "avg_copies" : {
            "value" : 3.5
          }
        },
        "available" : {
          "doc_count" : 2,
          "avg_copies" : {
            "value" : 0.5
          }
        }
      }
    }
  }
}

As you can see, we got two buckets, which is what we expected.

One of the most commonly used bucket aggregations is the terms aggregation. It allows us to get information about the terms and the count of documents having those terms. For example, one of the simplest uses is getting the count of the books that are available and not available. We can do that by running the following query:

{
 "aggs" : {
  "counts" : {
   "terms" : {
    "field" : "available"
   }
  }
 }
}

In the response, we will get two buckets (because the Boolean field can only have two values – true and false). Here, this will look as follows:

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "counts" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ {
        "key" : 0,
        "key_as_string" : "false",
        "doc_count" : 2
      }, {
        "key" : 1,
        "key_as_string" : "true",
        "doc_count" : 2
      } ]
    }
  }
}

By default, the data is sorted on the basis of document count, which means that the most common terms will be placed on top of the aggregation results. Of course, we can control this behavior by specifying the order property and providing the order just like we usually do when sorting by arbitrary field values. Elasticsearch allows us to sort by the document count (using the _count static value) and by the term (using the _term static value). For example, if we want to sort our preceding aggregation results by descending term, we can run the following query:

{
 "aggs" : {
  "counts" : {
   "terms" : {
    "field" : "available",
    "order" : { "_term" : "desc" }   }
  }
 }
}

However, that's not all when it comes to sorting. We can also sort by the results of the nested aggregations that were included in the query.

{
 "aggs" : {
  "counts" : {
   "terms" : {
    "field" : "available",
    "min_doc_count" : 5   }
  }
 }
}

The range aggregation allows us to define one or more ranges and Elasticsearch calculates buckets for them. For example, if we want to check how many books were published in a given period of time, we create the following query:

{
 "aggs": {
  "years": {
   "range": {
    "field": "year",
    "ranges": [
     { "to" : 1850 },
     { "from": 1851, "to": 1900 },
     { "from": 1901, "to": 1950 },
     { "from": 1951, "to": 2000 },
     { "from": 2001 }
    ]
   }
  }
 }
}

We specify the field we want the aggregation to be calculated on and the array of ranges. Each range is defined by one or two properties: the two and from similar to the range queries which we already discussed.

The result returned by Elasticsearch for our data looks as follows:

{
  "took" : 23,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "years" : {
      "buckets" : [ {
        "key" : "*-1850.0",
        "to" : 1850.0,
        "to_as_string" : "1850.0",
        "doc_count" : 0
      }, {
        "key" : "1851.0-1900.0",
        "from" : 1851.0,
        "from_as_string" : "1851.0",
        "to" : 1900.0,
        "to_as_string" : "1900.0",
        "doc_count" : 1
      }, {
        "key" : "1901.0-1950.0",
        "from" : 1901.0,
        "from_as_string" : "1901.0",
        "to" : 1950.0,
        "to_as_string" : "1950.0",
        "doc_count" : 2
      }, {
        "key" : "1951.0-2000.0",
        "from" : 1951.0,
        "from_as_string" : "1951.0",
        "to" : 2000.0,
        "to_as_string" : "2000.0",
        "doc_count" : 1
      }, {
        "key" : "2001.0-*",
        "from" : 2001.0,
        "from_as_string" : "2001.0",
        "doc_count" : 0
      } ]
    }
  }
}

For example, between 1901 and 1950 we had two books released.

{
 "aggs": {
  "years": {
   "range": {
    "field": "year",
    "keyed": true,
    "ranges": [
     { "key": "Before 18th century", "to": 1799 },
     { "key": "18th century", "from": 1800, "to": 1899 },
     { "key": "19th century", "from": 1900, "to": 1999 },
     { "key": "After 19th century", "from": 2000 }
    ]
   }
  }
 }
}

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "years" : {
      "buckets" : {
        "Before 18th century" : {
          "to" : 1799.0,
          "to_as_string" : "1799.0",
          "doc_count" : 0
        },
        "18th century" : {
          "from" : 1800.0,
          "from_as_string" : "1800.0",
          "to" : 1899.0,
          "to_as_string" : "1899.0",
          "doc_count" : 1
        },
        "19th century" : {
          "from" : 1900.0,
          "from_as_string" : "1900.0",
          "to" : 1999.0,
          "to_as_string" : "1999.0",
          "doc_count" : 3
        },
        "After 19th century" : {
          "from" : 2000.0,
          "from_as_string" : "2000.0",
          "doc_count" : 0
        }
      }
    }
  }
}

The date_range aggregation is similar to the previously discussed range aggregation but it is designed for fields that use date-based types. However, in the library index, the documents have years, but the field is a number, not a date. For the purpose of showing how this aggregation works, let's imagine that we want to extend our library index to support newspapers. To do this we will create a new index (called library2) by using the following command:

curl -XPOST localhost:9200/_bulk --data-binary '{ "index": {"_index": "library2", "_type": "book", "_id": "1"}}
{ "title": "Fishing news", "published": "2010/12/03 10:00:00", "copies": 3, "available": true }
{ "index": {"_index": "library2", "_type": "book", "_id": "2"}}
{ "title": "Knitting magazine", "published": "2010/11/07 11:32:00", "copies": 1, "available": true }
{ "index": {"_index": "library2", "_type": "book", "_id": "3"}}
{ "title": "The guardian", "published": "2009/07/13 04:33:00", "copies": 0, "available": false }
{ "index": {"_index": "library2", "_type": "book", "_id": "4"}}
{ "title": "Hadoop World", "published": "2012/01/01 04:00:00", "copies": 6, "available": true }
'

For the purpose of this example, we will leave the mappings definition for Elasticsearch – this is sufficient in this case. Let's start with the first query using the date_range aggregation:

{
 "aggs": {
  "years": {
   "date_range": {
    "field": "published",
    "ranges": [
     { "to" : "2009/12/31" },
     { "from": "2010/01/01", "to": "2010/12/31" },
     { "from": "2011/01/01" }
    ]
   }
  }
 }
}

Compared with the ordinary range aggregation, the only thing that changed is the aggregation type, which is now date_range. The dates can be passed as a string in a form recognized by Elasticsearch or as a number value (number of milliseconds since 1970-01-01). The response returned by Elasticsearch for the preceding query looks as follows:

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "years" : {
      "buckets" : [ {
        "key" : "*-2009/12/31 00:00:00",
        "to" : 1.2622176E12,
        "to_as_string" : "2009/12/31 00:00:00",
        "doc_count" : 1
      }, {
        "key" : "2010/01/01 00:00:00-2010/12/31 00:00:00",
        "from" : 1.262304E12,
        "from_as_string" : "2010/01/01 00:00:00",
        "to" : 1.2937536E12,
        "to_as_string" : "2010/12/31 00:00:00",
        "doc_count" : 2
      }, {
        "key" : "2011/01/01 00:00:00-*",
        "from" : 1.29384E12,
        "from_as_string" : "2011/01/01 00:00:00",
        "doc_count" : 1
      } ]
    }
  }
}

As you can see, the response is no different when compared to the response returned by the range aggregation. We have two attributes for each bucket - named from and to which represent the number of milliseconds from 1970-01-01. The properties from_as_string and to_as_string present the same information as from and to, but in a human-readable form. Of course the keyed parameter and key in the definition of date range work in the already described way.

Elasticsearch also allows us to define the format of presented dates using the format attribute. In our example, we presented the dates with year resolution, so the day and time parts were unnecessary. If we want to show the month names, we can send a query such as the following one:

{
 "aggs": {
  "years": {
   "date_range": {
    "field": "published",
    "format": "MMMM YYYY",
    "ranges": [
     { "to" : "December 2009" },
     { "from": "January 2010", "to": "December 2010" },
     { "from": "January 2011" }
    ]
   }
  }
 }
}

Note that the dates in the to and from parameters also need to be provided in the specified format. One of the returned ranges looks as follows:

{
 "key" : "January 2010-December 2010",
 "from" : 1.262304E12,
 "from_as_string" : "January 2010",
 "to" : 1.2911616E12,
 "to_as_string" : "December 2010",
 "doc_count" : 1
}

There is one more thing about the date_range aggregation that we want to mention. Imagine that some time we may want to build an aggregation that can change with time. For example, we may want to see how many newspapers were published in the last 3, 6, 9, and 12 months. This is possible without the need to adjust the query every time, as we can use constants such as now-9M. The following example shows this:

{
 "aggs": {
  "years": {
   "date_range": {
    "field": "published",
    "format": "dd-MM-YYYY",
    "ranges": [
     { "to" : "now-9M/M"  },
     { "to" : "now-9M"  },
     { "from": "now-6M/M", "to": "now-9M/M" },
     { "from": "now-3M/M" }
    ]
   }
  }
 }
}

The key here is expressions such as now-9M. Elasticsearch does the math and generates the appropriate value. For example, you can use y (year), M (month), w (week), d (day), h (hour), m (minute), and s (second). For example, the expression now+3d means three days from now. The /M in our example takes only the date rounded to months. Thanks to such notation, we only count full months. The second advantage is that the calculated date is more cache-friendly without the rounding date changes every millisecond that make every cache based on the range irrelevant and basically useless in most cases.

A very interesting aggregation is the ip_range one as it works on Internet addresses. It works on the fields defined with the ip type and allows defining ranges given by the IP range in CIDR notation (http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing). An example usage of the ip_range aggregation looks as follows:

{
 "aggs": {
  "access": {
   "ip_range": {
    "field": "ip",
    "ranges": [
     { "from": "192.168.0.1", "to": "192.168.0.254" },
     { "mask": "192.168.1.0/24" }
    ]
   }
  }
 }
}

The response to the preceding query is as follows:

      "access": {
         "buckets": [
            {
               "from": 3232235521,
               "from_as_string": "192.168.0.1",
               "to": 3232235774,
               "to_as_string": "192.168.0.254",
               "doc_count": 0
            },
            {
               "key": "192.168.1.0/24",
               "from": 3232235776,
               "from_as_string": "192.168.1.0",
               "to": 3232236032,
               "to_as_string": "192.168.2.0",
               "doc_count": 4
            }
         ]
      }

Similar to the range aggregation, we define both ends of the brackets and the mask. The rest is done by Elasticsearch itself.

The missing aggregation allows us to create a bucket and see how many documents have no value in a specified field. For example, we can check how many of our books in the library index don't have the original title defined – the otitle field. To do this, we run the following query:

{
 "aggs": {
  "missing_original_title": {
   "missing": {
    "field": "otitle"
   }
  }
 }
}

The response returned by Elasticsearch in this case will look as follows:

{
  "took" : 15,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "missing_original_title" : {
      "doc_count" : 2
    }
  }
}

As we can see, we have two documents without the otitle field.

The histogram aggregation is an interesting one because of its automation. This aggregation defines buckets itself. We are only responsible for defining the field and the interval, and the rest is done automatically. The simplest form of a query that uses this aggregation looks as follows:

{
 "aggs": {
  "years": {
   "histogram": {
    "field" : "year",
    "interval": 100
   }
  }
 }
}

The new information we need to provide is interval, which defines the length of every range that will be used to create a bucket. We set the interval to 100, which in our case will result in buckets that are 100 years wide. The aggregation part of the response to the preceding query that was sent to our library index is as follows:

{
  "took" : 13,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "years" : {
      "buckets" : [ {
        "key" : 1800,
        "doc_count" : 1
      }, {
        "key" : 1900,
        "doc_count" : 3
      } ]
    }
  }
}

Similar to the range aggregation, the histogram aggregation allows us to use the keyed property to define named buckets. The other available option is min_doc_count, which allows us to specify the minimum number of documents required to create a bucket. If we set the min_doc_count property to zero, Elasticsearch will also include buckets with the document count of zero. We can also use the missing property to specify the value Elasticsearch should use when a document doesn't have a value in the specified field.

Elasticsearch stores all the dates in the UTC time zone. You can define the time zone to be used by Elasticsearch by using the time_zone attribute. By setting this property, we basically tell Elasticsearch which time zone should be used to perform the calculations. There are three notations with which to set these attributes:

To calculate significant terms, Elasticsearch looks for data that reports a significant change in their popularity between two sets of data: the foreground set and the background set. The foreground set is the data returned by our query, while the background set is the data in our index (or indices, depending on how we run our queries). If a term exists in 10 documents out of one million indexed, but appears in 5 documents from the 10 returned, then such a term is definitely significant and worth concentrating on.

Let's get back to our preceding example now to analyze it a bit. Richard got three grades from the reviewers – bad three times, medium one time, and good one time. From these three, the bad value appeared in three out of the five documents matching the query. In general, the bad grade appeared in three documents (the bg_count property) out of the 12 documents in the index (this is our background set). This gives us 25 percent of the indexed documents. On the other hand, the bad grade appeared in three out of the five documents matching the query (this is our foreground set), which gives us 60 percent of the documents. As you can see, the change in popularity is significant for the bad grade and that's why Elasticsearch has returned it in the significant_terms aggregation results.

The significant_terms aggregation can be nested and provide us with nice data analysis capabilities that connect two multiple sets of data. For example, let's try to find a significant grade for each of the interns that we have information about. To do this we will nest the significant_terms aggregation inside the terms aggregation. The query that does that looks as follows:

curl -XGET 'localhost:9200/interns/_search?size=0&pretty' -d '{
 "aggregations" : {
  "grades" : {
   "terms" : {
    "field" : "intern"
   },
   "aggregations" : {
    "significantGrades" : {
     "significant_terms" : {
      "field" : "grade"
     }
    }
   }
  }
 }
}'

The results returned by Elasticsearch for the preceding query are as follows:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 12,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "grades" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ {
        "key" : "ralf",
        "doc_count" : 5,
        "significantGrades" : {
          "doc_count" : 5,
          "buckets" : [ {
            "key" : "good",
            "doc_count" : 3,
            "score" : 0.48,
            "bg_count" : 4
          } ]
        }
      }, {
        "key" : "richard",
        "doc_count" : 5,
        "significantGrades" : {
          "doc_count" : 5,
          "buckets" : [ {
            "key" : "bad",
            "doc_count" : 3,
            "score" : 0.84,
            "bg_count" : 3
          } ]
        }
      }, {
        "key" : "monica",
        "doc_count" : 2,
        "significantGrades" : {
          "doc_count" : 2,
          "buckets" : [ ]
        }
      } ]
    }
  }
}

There is one more feature about nested aggregations and the ordering of aggregation results. Elasticsearch can use values from the nested aggregations to sort the parent buckets. For example, let's look at the following query:

{
 "aggs": {
  "availability": {
   "terms": {
    "field": "copies",
    "order": { "numbers.avg": "desc" }
   },
   "aggs": {
    "numbers": { "stats" : {} }
   }
  }
 }
}

In the previous example, the order in the availability aggregation is based on the average value from the numbers aggregation. The notation numbers.avg is required in this case, because stats is a multivalued aggregation and provides multiple information and we were interested in the average. If it were the sum aggregation, the name of the aggregation would be sufficient.