The first solution is to completely disable the schema-less behavior in Elasticsearch. We can do that by adding the index.mapper.dynamic property to our index properties and setting it to false. We can do that by running the following command to create the index:

curl -XPUT 'localhost:9200/sites' -d '{
  "index.mapper.dynamic": false
}'

By doing that we told Elasticsearch that we don't want it to guess the type of our documents in the site's index and that we will provide the mappings ourselves. If we will try indexing some example document to the site's index, we will get the following error:

{
  "error" : {
    "root_cause" : [ {
      "type" : "type_missing_exception",
      "reason" : "type[[doc, trying to auto create mapping, but dynamic mapping is disabled]] missing",
      "index" : "sites"
    } ],
    "type" : "type_missing_exception",
    "reason" : "type[[doc, trying to auto create mapping, but dynamic mapping is disabled]] missing",
    "index" : "sites"
  },
  "status" : 404
}

This is because we didn't create any mappings – no schema for documents was created. Elasticsearch couldn't create one for us because we didn't allow it and the indexation command failed.

Of course this is not the only thing we can do when it comes to configuring how the type determining mechanism works. We can also tune it or disable it for a given type on the object level. We will talk about the second case in Chapter 5, Extending Your Index Structure. For now, let's look at the possibilities of tuning type determining mechanism in Elasticsearch.

One of the solutions to the problems with JSON documents and type guessing is that we are not always in control of the data. The documents that we are indexing can come from multiple places and some systems in our environment may include quotation marks for all the fields in the document. This can lead to problems and bad guesses. Because of that, Elasticsearch allows us to enable more aggressive fields value checking for numeric fields by setting the numeric_detection property to true in the mappings definition. For example, let's assume that we want to create an index called users and we want it to have the user type on which we will want more aggressive numeric fields parsing. To do that, we will use the following command:

curl -XPUT http://localhost:9200/users/?pretty -d '{ 
  "mappings" : {
    "user": {
      "numeric_detection" : true
    }
  }
}'

Now let's run the following command to index a single document to the users index:

curl -XPOST http://localhost:9200/users/user/1 -d '{"name": "User 1", "age": "20"}'

Earlier, with the default settings, the age field would be set to string type. With the numeric_detection property set to true, the type of the age field will be set to long. We can check that by running the following command (it will retrieve the mappings for all the types in the users index):

curl -XGET 'localhost:9200/users/_mapping?pretty'

The preceding command should result in the following response returned by Elasticsearch:

{
  "users" : {
    "mappings" : {
      "user" : {
        "numeric_detection" : true,
        "properties" : {
          "age" : {
            "type" : "long"
          },
          "name" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

As we can see, the age field was really set to be of type long.

Another type of data that causes trouble are fields with dates. Dates can come in different flavors, for example, 2015-10-01 11:22:33 is a proper date and so is 2015-10-01T11:22:33+00. Because of that, Elasticsearch tries to match the fields to timestamps or strings that match some given date format. If that matching operation is successful, the field is treated as a date based one. If we know how our date fields look, we can help Elasticsearch by providing a list of recognized date formats using the dynamic_date_formats property, which allows us to specify the formats array. Let's look at the following command for creating an index:

curl -XPUT 'http://localhost:9200/blog/' -d '{ 
  "mappings" : {
    "article" : {
      "dynamic_date_formats" : ["yyyy-MM-dd hh:mm"]
    }
  }
}'

The preceding command will result in the creation of an index called blog with the single type called article. We've also used the dynamic_date_formats property with a single date format that will result in Elasticsearch using the date core type (refer to the Core types section in this chapter for more information about field types) for fields matching the defined format. Elasticsearch uses the joda-time library to define the date formats, so visit http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html if you are interested in knowing about them.

With the preceding index, we can now try indexing a new document using the following command:

curl -XPUT localhost:9200/blog/article/1 -d '{"name": "Test", "test_field":"2015-10-01 11:22"}'

Elasticsearch will of course index that document, but let's look at the mappings created for our index:

curl -XGET 'localhost:9200/blog/_mapping?pretty'

The response for the preceding command will be as follows:

{
  "blog" : {
    "mappings" : {
      "article" : {
        "dynamic_date_formats" : [ "yyyy-MM-dd hh:mm" ],
        "properties" : {
          "name" : {
            "type" : "string"
          },
          "test_field" : {
            "type" : "date",
            "format" : "yyyy-MM-dd hh:mm"
          }
        }
      }
    }
  }
}

As we can see, the test_field field was given a date type, so our tuning works.

Unfortunately, the problem still exists if we want the Boolean type to be guessed. There is no option to force the guessing of Boolean types from the text. In such cases, when a change of source format is impossible, we can only define the field directly in the mappings definition.

The mappings definition in Elasticsearch is just another JSON object, so it needs to be properly started and ended with curly brackets. All the mappings definitions are nested inside a single mappings object. In our example, we had a single post type, but we can have multiple of them. For example, if we would like to have more than a single type in our mappings, we just need to separate them with a comma character. Let's assume that we would like to have an additional user type in our posts index. The mappings definition in such case will look as follows (we stored it in the posts_with_user.json file):

{
  "mappings": {
    "post": {
      "properties": {
        "id": { "type":"long" },
        "name": { "type":"string" },
        "published": { "type":"date" },
        "contents": { "type":"string" }
      }
    },
    "user": {
      "properties": {
        "id": { "type":"long" },
        "name": { "type":"string" }
      }
    }
  }
}

As you can see, we can name the types with the names we want. Under each type we have the properties object in which we store the actual name of the fields and their definition.

Each field in the mappings definition is just a name and an object describing the properties of the field. For example, we can have a field defined as the following:

"body": { "type":"string", "store":"yes", "index":"analyzed" }

The preceding field definition starts with a name – body. After that we have an object with three properties – the type of the field (the type property), if the original field value should be stored (the store property), and if the field should be indexed and how (the index property). And, of course, multiple field definitions are separated from each other using the comma character, just like other JSON objects.

Each field type in Elasticsearch can be given one of the provided core types. The core types in Elasticsearch are as follows:

In addition to the core types, Elasticsearch provides additional types that can handle more complicated data – such as nested documents, object, and so on. We will talk about them in Chapter 5, Extending Your Index Structure.

"body" : { "type" : "string", "store" : "yes", "index" : "analyzed" }

"price" : { "type" : "float", "precision_step" : "4" }

"allowed" : { "type" : "boolean", "store": "yes" }

"image" : { "type" : "binary" }

"published" : { "type" : "date", "format" : "YYYY-mm-dd" }

{
  "name" : "Sample document",
  "published" : "2012-12-22"
}

There are situations where we need to have the same field analyzed differently. For example, one for sorting, one for searching, and one for analysis with aggregations, but all using the same field value, just indexed differently. We could of course use the previously described field value copying, but we can also use so called multi fields. To be able to use that feature of Elasticsearch, we need to define an additional property in our field definition called fields. The fields is an object that can contain one or more additional fields that will be present in our index and will have the value of the field that they are assigned to. For example, if we would like to have aggregations done on the name field and in addition to that search on that field, we would define it as follows:

"name": {
  "type": "string",
  "fields": {
    "agg": { "type" : "string", "index": "not_analyzed" }
  }
}

The preceding definition will create two fields – one called name and the second called name.agg. Of course, you don't have to specify two separate fields in the data you are sending to Elasticsearch – a single one named name is enough. Elasticsearch will do the rest, which means copying the value of the field to all the fields from the preceding definition.

The ip field type was added to Elasticsearch to simplify the use of IPv4 addresses in a numeric form. This field type allows us to search data that is indexed as an IP address, sort on such data, and use range queries using IP values.

A sample definition of a field based on one of the numeric types is as follows:

"address" : { "type" : "ip" }

In addition to the common attributes, the precision_step attribute can also be set for the ip type based fields. Refer to the numeric type description for more information about that property.

A sample document that uses the ip based field looks as follows:

{
  "name" : "Tom PC",
  "address" : "192.168.2.123"
}

The token_count field type allows us to store and index information about how many tokens the given field has instead of storing and indexing the text provided to the field. It accepts the same configuration options as the number type, but in addition to that, we need to specify the analyzer which will be used to divide the field value into tokens. We do that by using the analyzer property.

A sample definition of a field based on the token_count field type looks as follows:

"title_count" : { "type" : "token_count", "analyzer" : "standard" }

Elasticsearch allows us to use one of the many analyzers defined by default. The following analyzers are available out of the box:

In addition to the analyzers mentioned previously, Elasticsearch allows us to define new ones without the need for writing a single line of Java code. In order to do that, we need to add an additional section to our mappings file; that is, the settings section, which holds additional information used by Elasticsearch during index creation. The following code snippet shows how we can define our custom settings section:

"settings" : {
  "index" : {
    "analysis": {
      "analyzer": {
        "en": {
          "tokenizer": "standard",
          "filter": [
            "asciifolding",
            "lowercase",
            "ourEnglishFilter"
          ]
        }
      },
      "filter": {
        "ourEnglishFilter": {
          "type": "kstem"
        }
      }
    }
  }
}

We specified that we want a new analyzer named en to be present. Each analyzer is built from a single tokenizer and multiple filters. A complete list of the default filters and tokenizers can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenizers.html. Our en analyzer includes the standard tokenizer and three filters: asciifolding and lowercase, which are the ones available by default, and a custom ourEnglishFilter, which is a filter we have defined.

To define a filter, we need to provide its name, its type (the type property), and any number of additional parameters required by that filter type. The full list of filter types available in Elasticsearch can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenfilters.html. Please be aware, that we won't be discussing each filter as the list of filters is constantly changing. If you are interested in the full filters list, please refer to the mentioned page in the documentation.

So, the final mappings file with our custom analyzer defined will be as follows:

{
  "settings" : {
    "index" : {
      "analysis": {
        "analyzer": {
          "en": {
            "tokenizer": "standard",
            "filter": [
             "asciifolding",
             "lowercase",
             "ourEnglishFilter"
            ]
          }
        },
        "filter": {
          "ourEnglishFilter": {
            "type": "kstem"
          }
        }
      }
    }
  },
  "mappings" : {
    "post" : {
      "properties" : { 
        "id": { "type" : "long" },
        "name": { "type" : "string", "analyzer": "en" } 
      }
    }
  }
}

If we save the preceding mappings to a file called posts_mappings.json, we can run the following command to create the posts index:

curl -XPOST 'http://localhost:9200/posts' -d @posts_mappings.json

We can see how our analyzer works by using the Analyze API (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html). For example, let's look at the following command:

curl -XGET 'localhost:9200/posts/_analyze?pretty&field=name' -d 'robots cars'

The command asks Elasticsearch to show the content of the analysis of the given phrase (robots cars) with the use of the analyzer defined for the post type and its name field. The response that we will get from Elasticsearch is as follows:

{
  "tokens" : [ {
    "token" : "robot",
    "start_offset" : 0,
    "end_offset" : 6,
    "type" : "<ALPHANUM>",
    "position" : 0
  }, {
    "token" : "car",
    "start_offset" : 7,
    "end_offset" : 11,
    "type" : "<ALPHANUM>",
    "position" : 1
  } ]
}

As you can see, the robots cars phrase was divided into two tokens. In addition to that, the robots word was changed to robot and the cars word was changed to car.

There is one more thing to say about analyzers. Elasticsearch allows us to specify the analyzer that should be used by default if no analyzer is defined. This is done in the same way as we configured a custom analyzer in the settings section of the mappings file, but instead of specifying a custom name for the analyzer, a default keyword should be used. So to make our previously defined analyzer the default, we can change the en analyzer to the following:

{
  "settings" : {
    "index" : {
      "analysis": {
        "analyzer": {
          "default": {
            "tokenizer": "standard",
            "filter": [
             "asciifolding",
             "lowercase",
             "ourEnglishFilter"
            ]
          }
        },
        "filter": {
          "ourEnglishFilter": {
            "type": "kstem"
          }
        }
      }
    }
  }
}

We can also choose a different default analyzer for searching and a different one for indexing. If we would like to do that instead of using the default keyword for the analyzer name, we should use default_search and default_index respectively.

Since Elasticsearch 0.90, we are allowed to set a different similarity for each of the fields that we have in our mappings file. For example, let's assume that we have the following simple mappings that we use in order to index the blog posts:

{
  "mappings" : {
    "post" : {
      "properties" : {
        "id" : { "type" : "long" },
        "name" : { "type" : "string" },
        "contents" : { "type" : "string" }
      }
    }
  }
}

To do this, we will use the BM25 similarity model for the name field and the contents field. In order to do that, we need to extend our field definitions and add the similarity property with the value of the chosen similarity name. Our changed mappings will look like the following:

{
  "mappings" : {
    "post" : {
      "properties" : {
        "id" : { "type" : "long" },
        "name" : { "type" : "string", "similarity" : "BM25" },
        "contents" : { "type" : "string", "similarity" : "BM25" }
      }
    }
  }
}

And that's all, nothing more is needed. After the above change, Apache Lucene will use the BM25 similarity to calculate the score factor for the name and the contents fields.

There are at least five new similarity models available. For most of the use cases, apart from the default one, you may find the following models useful:

The two other similarity models currently available are LM Dirichlet similarity (to use it, set the type property to LMDirichlet) and LM Jelinek Mercer similarity (to use it, set the type property to LMJelinekMercer). You can find more about these similarity models in Apache Lucene Javadocs, Mastering Elasticsearch Second Edition, published by Packt Publishing or in official documentation of Elasticsearch available at https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html.

curl -XPUT 'localhost:9200/test_similarity' -d '{
 "settings" : {
  "similarity" : {
   "altered_default": {
    "type" : "default",
    "discount_overlaps" : false
   }
  }
 },
 "mappings": {
  "doc": {
   "properties": {
    "name": { "type" : "string", "similarity": "altered_default" }
   }
  }
 }
}'

      "similarity" : {
        "esserverbook_dfr_similarity" : {
          "type" : "DFR",
          "basic_model" : "g",
          "after_effect" : "l",
          "normalization" : "h2",
          "normalization.h2.c" : "2.0"
        }
      }

      "similarity" : {
        "esserverbook_ib_similarity" : {
          "type" : "IB",
          "distribution" : "ll",
          "lambda" : "df",
          "normalization" : "z",
          "normalization.z.z" : "0.25"
        }
      }