In the Mappings configuration section of Chapter 2, Indexing Your Data, we discussed mappings, how they are created, and how the type-determining mechanism works. Now we will get into more advanced topics. We will show you how to dynamically create mappings for new indices and how to apply some logic to the templates, so that new indices are already created with predefined mappings.
In various parts of the book, when discussing index configuration and its structure, we've seen that this can become complicated, especially when we have sophisticated data structures that we want to index, search, and aggregate. Especially if you have a lot of similar indices, taking care of the mappings in each of them can be a very painful process – each new index has to be created with appropriate mappings. Elasticsearch creators predicted this and implemented a feature called index templates. Each template defines a pattern, which is compared to a newly created index name. When both of them match, the values defined in the template are copied to the index structure definition. When multiple templates match the name of the newly created index, all of them are applied and the values from the templates that are applied later override the values defined in the previously applied templates. This is very convenient because we can define a few common settings in the general templates and change them in the more specialized ones. In addition, there is an order parameter that lets us force the desired template ordering. You can think of templates as dynamic mappings that can be applied not to the types in documents but to the indices.
Let's see a real example of a template. Imagine that we want to create many indices in which we don't want to store the source of the documents so that our indices are smaller. We also don't need any replicas. We can create a template that matches our need by using the Elasticsearch REST API and the /_template end point, by sending the following command:
curl -XPUT http://localhost:9200/_template/main_template?pretty -d '{ "template" : "*", "order" : 1, "settings" : { "index.number_of_replicas" : 0 }, "mappings" : { "_default_" : { "_source" : { "enabled" : false } } } }'
From now on, all the created indices will have no replicas and no source stored. This is because the template parameter value is set to *, which matches all the names of the indices. Note the _default_ type name in our example. This is a special type name which indicates that the current rule should be applied to every document type. The second interesting thing is the order parameter. Let's define a second template by using the following command:
curl -XPUT http://localhost:9200/_template/ha_template?pretty -d '{ "template" : "ha_*", "order" : 10, "settings" : { "index.number_of_replicas" : 5 } }'
After running the preceding command, all the new indices will behave as earlier except the ones with names beginning with ha_. In case of these indices, both the templates are applied. First, the template with the lower order value is used and then the next template overwrites the replica's setting. So, the indices whose names start with ha_ will have five replicas and disabled sources stored.
Sometimes we want to have the possibility of defining type that is dependent on the field name and the type. This is where dynamic templates can help. Dynamic templates are similar to the usual mappings, but each template has its pattern defined, which is applied to a document's field name. If a field name matches the pattern, the template is used.
Let's have a look at the following example:
curl -XPOST 'localhost:9200/news' -d '{ "mappings" : { "article" : { "dynamic_templates" : [ { "template_test": { "match" : "*", "mapping" : { "index" : "analyzed", "fields" : { "str": { "type": "{dynamic_type}", "index": "not_analyzed" } } } } } ] } } }'
In the preceding example, we defined the mapping for the article type. In this mapping, we have only one dynamic template named template_test. This template is applied for every field in the input document because of the single asterisk pattern in the match property. Each field will be treated as a multifield, consisting of a field named as the original field (for example, title) and the second field with a name suffixed with str (for example, title.str). The first field will have its type determined by Elasticsearch (with the {dynamic_type} type), and the second field will be a string (because of the string type).
We have two ways of defining the matching pattern. They are as follows:
By default, the pattern is very simple and uses glob patterns. This can be changed by using match_pattern=regexp. After adding this property, we can use all the magic provided by regular expressions to match and unmatch the patterns. There are variations such as path_match and path_unmatch that can be used to match the names in nested documents (by providing path, similar to queries).