The Elasticsearch term suggester supports multiple configuration properties that allow us to tune its behavior to match our needs and to work with our data. Of course, we've already seen how it works and what it can give us, so we will concentrate on configuration now.

The common term suggester options can be used for all the suggester implementations that are based on the term suggester. Currently, these are the phrase suggester and, of course, the base term suggester. The available options are:

In addition to the common term suggester options, Elasticsearch allows us to use additional ones that will only make sense for the term suggester itself. These options are as follows:

Before we continue with all the possibilities, we have to configure the phrase suggester; let's start with showing you an example of how to use it. This time, we will run a simple query to the _search endpoint with only the suggests section in it. We do this by running the following command:

curl -XGET 'localhost:9200/wikipedia/_search?pretty' -d '{
 "suggest" : {
  "text" : "wordl war ii",
  "our_suggestion" : {
   "phrase" : {
    "field" : "_all"
   }
  }
 }
}'

As you can see in the preceding command, it is almost the same as we sent when using the term suggester, but instead of specifying the term suggester type, we've specified the phrase type. The response to the preceding command will be as follows:

{
  "took" : 58,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 7080049,
    "max_score" : 1.0,
    "hits" : [
    ...
    ]
  },
  "suggest" : {
    "our_suggestion" : [ {
      "text" : "wordl war ii",
      "offset" : 0,
      "length" : 12,
      "options" : [ {
        "text" : "world war ii",
        "score" : 7.055394E-5
      }, {
        "text" : "words war ii",
        "score" : 2.3738032E-5
      }, {
        "text" : "wordy war ii",
        "score" : 3.575829E-6
      }, {
        "text" : "worde war ii",
        "score" : 1.1586584E-6
      }, {
        "text" : "woudl war ii",
        "score" : 1.0753317E-6
      } ]
    } ]
  }
}

As you can see, the response is very similar to the one returned by the term suggester, but instead of a single word being returned as the suggestion for each term from the text field, it is already combined and Elasticsearch returns whole phrases. Of course, we can configure additional parameters in the phrase section and, now, we will look at what parameters are available for usage. Of course, the returned suggestions are sorted by their score by default.

The phrase suggester configuration parameter can be divided into three groups: basic parameters that define the general behavior, the smoothing models configuration to balance N-grams' weights, and candidate generators that are responsible for producing the list of terms suggestions that will be used to return final suggestions.

In addition to properties mentioned in the preceding phrase, the suggester exposes the following basic options:

Let's now look at an example of using some of the preceding mentioned parameters, for example, suggestions highlighting. If we modify our initial phrase suggestion query and add highlighting, the command would look as follows:

curl -XGET 'localhost:9200/wikipedia/_search?pretty' -d '{
 "suggest" : {
  "text" : "wordl war ii",
  "our_suggestion" : {
   "phrase" : {
    "field" : "_all", 
    "highlight" : {
     "pre_tag" : "<b>",
     "post_tag" : "</b>"
    },
    "collate" : {
     "prune" : true,
     "query" : {
      "match" : {
       "title" : "{{suggestion}}"
      }
     }
    }
   }
  }
 }
}'

The result returned by Elasticsearch for the preceding query would be as follows:

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 7080049,
    "max_score" : 1.0,
    "hits" : [
    ...
    ]
  },
    "suggest" : {
    "our_suggestion" : [ {
      "text" : "wordl war ii",
      "offset" : 0,
      "length" : 12,
      "options" : [ {
        "text" : "world war ii",
        "highlighted" : "<b>world</b> war ii",
        "score" : 7.055394E-5,
        "collate_match" : true
      }, {
        "text" : "words war ii",
        "highlighted" : "<b>words</b> war ii",
        "score" : 2.3738032E-5,
        "collate_match" : true
      }, {
        "text" : "wordy war ii",
        "highlighted" : "<b>wordy</b> war ii",
        "score" : 3.575829E-6,
        "collate_match" : true
      }, {
        "text" : "worde war ii",
        "highlighted" : "<b>worde</b> war ii",
        "score" : 1.1586584E-6,
        "collate_match" : true
      }, {
        "text" : "woudl war ii",
        "highlighted" : "<b>woudl</b> war ii",
        "score" : 1.0753317E-6,
        "collate_match" : true
      } ]
    } ]
  }
}

As you can see, the suggestions were highlighted.

A Smoothing model is a functionality of the phrase suggester whose responsibility is to measure the balance between the weight of infrequent N-grams that don't exist in the index and the frequent ones that exist in the index. It is rather an expert option and if you want to modify these N-grams, you should check suggester responses for your queries in order to see whether your suggestions are proper for your case. Smoothing is used in language models to avoid situations where the probability of a given term is equal to zero. The Elasticsearch phrase suggester supports multiple smoothing models.

In order to set which smoothing model we want to use, we need to add an object called smoothing and include a smoothing model name we want to use inside of it. Of course, we can include the properties we need or want to set for the given smoothing model. For example, we could run the following command:

curl -XGET 'localhost:9200/wikipedia/_search?pretty&size=0' -d '{
 "suggest" : {
  "text" : "wordl war ii",
  "generators_example_suggestion" : {
   "phrase" : {
    "analyzer" : "standard",
    "field" : "_all",
    "smoothing" : {
     "linear" : {
      "trigram_lambda" : 0.1,
      "bigram_lambda" : 0.6,
      "unigram_lambda" : 0.3
     }
    }
   }
  }
 }
}'

There are three smoothing models available in Elasticsearch. Let's now look at them.

Stupid backoff is the default smoothing model used by the Elasticsearch phrase suggester. In order to alter it or force its usage, we need to use the stupid_backoff name. The stupid backoff smoothing model is an implementation that will use a lower ordered N-gram (and will give it a discount equal to the value of the discount property) if the higher order N-gram count is equal to 0. To illustrate the example, let's assume that we use the ab bigram and the c unigram, which are common and exist in our index used by the suggester. However, we don't have the abc trigram present. What the stupid backoff model will do is that it will use the ab bigram model, because abc doesn't exist and, of course, the ab bigram model will be given a discount equal to the value of the discount property.

The stupid backoff model provides a single property that we can alter: discount. By default, it is set to 0.4, and it is used as a discount factor for the lower ordered N-gram model.

You can read more about N-gram smoothing models by looking at http://en.wikipedia.org/wiki/N-gram#Smoothing_techniques and http://en.wikipedia.org/wiki/Katz's_back-off_model (which is similar to the stupid backoff model described).

The Laplace smoothing model is also called additive smoothing. When used (to use it, we need to use the laplace value as its name), a constant value equal to the value of the alpha parameter (which is by 0.5 default) will be added to counts to balance weights of frequent and infrequent N-grams. As mentioned, the Laplace smoothing model can be configured using the alpha property, which is set to 0.5 by default. The usual values for this parameter are typically equal or below 1.0.

You can read more about additive smoothing at http://en.wikipedia.org/wiki/Additive_smoothing.

Linear interpolation, the last smoothing model, takes the values of the lambdas provided in the configuration and uses them to calculate weights of trigrams, bigrams, and unigrams. In order to use the linear interpolation smoothing model, we need to provide the name of linear in the smoothing object in the suggester query and provide three parameters: trigram_lambda, bigram_lambda, and unigram_lambda. The sum of the values of the three mentioned parameters must be equal to 1. Each of these parameters is a weight for a given type of N-gram; for example, the bigram_lambda parameter value will be used as a weights for bigrams.

In order to return possible suggestions for a term from the text provided in the text parameter, Elasticsearch uses so-called candidate generators. You can think of candidate generators as term suggesters although they are not exactly the same—they are similar, because they are used for every single term in the query provided to suggester. After the candidate terms are returned, they are scored in combination with suggestions for other terms from the text, and this way, the phrase suggestions are built.

Currently, direct generators are the only candidate generators available in Elasticsearch, although we can expect more of them to be present in the future. Elasticsearch allows us to provide multiple direct generators in a single phrase suggester request. We can do this by providing the list named direct_generators. For example, we could run the following command:

curl -XGET 'localhost:9200/wikipedia/_search?pretty&size=0' -d '{
 "suggest" : {
  "text" : "wordl war ii",
  "generators_example_suggestion" : {
   "phrase" : {
    "analyzer" : "standard",
    "field" : "_all",
    "direct_generator" : [ 
     {
      "field" : "_all",
      "suggest_mode" : "always",
      "min_word_len" : 2
     }, 
     {
      "field" : "_all",
      "suggest_mode" : "always",
      "min_word_len" : 3
     } 
    ]
   }
  }
 }
}'

The response should be very similar to the one previously shown, so we decided to omit it.

Direct generators allow us to configure their behavior by using a parameter similar to that exposed by the terms suggester. These common configuration parameters are field (which is required), size, suggest_mode, max_edits, prefix_length, min_word_length (in this case, it defaults to 4), max_inspections, min_doc_freq, and max_term_freq. Refer to the term suggester description to see what these parameters mean.

In addition to the mentioned properties, direct generators allow us to use the pre_filter and post_filter properties. These two properties allow us to provide an analyzer name that Elasticsearch will use. The analyzer specified by the pre_filter property will be used for each term passed to the direct generator, and the filter specified by the post_filter property will be used after it is returned by the direct generator, just before these terms are passed to the phrase scorer for scoring.

For example, we could use the filtering functionality of the direct generators to include synonyms just before the suggestions are passed to the direct generator using the pre_filter property. For example, let's update our wikipedia index settings to include simple synonyms, and let's use them in filtering. To do this, we start with updating the settings with the following commands:

curl -XPOST 'localhost:9200/wikipedia/_close'
curl -XPUT 'localhost:9200/wikipedia/_settings' -d '{
 "settings" : {
  "index" : {
   "analysis": {
    "analyzer" : {
     "sample_synonyms_analyzer": {
      "tokenizer": "standard",
      "filter": [
       "sample_synonyms"
      ]
     }
    },
    "filter": {
     "sample_synonyms": {
      "type" : "synonym",
      "synonyms" : [
       "war => conflict"
      ]
     }
    }
   }         
  }
 }
}'
curl -XPOST 'localhost:9200/wikipedia/_open'

First, we need to close the index, update the setting, and then open it again because Elasticsearch won't allow us to change analysis settings on opened indices. Now we can test our direct generator with synonyms with the following command:

curl -XGET 'localhost:9200/wikipedia/_search?pretty&size=0' -d '{
 "suggest" : {
  "text" : "wordl war ii",
  "generators_with_synonyms" : {
   "phrase" : {
    "analyzer" : "standard",
    "field" : "_all",
    "direct_generator" : [ 
     {
      "field" : "_all",
      "suggest_mode" : "always",
      "post_filter" : "sample_synonyms_analyzer"
     }
    ]
   }
  }
 }
}'

The response to the preceding command should be as follows:

{
  "took" : 47,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 7080049,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "suggest" : {
    "generators_with_synonyms" : [ {
      "text" : "wordl war ii",
      "offset" : 0,
      "length" : 12,
      "options" : [ {
        "text" : "world war ii",
        "score" : 7.055394E-5
      }, {
        "text" : "words war ii",
        "score" : 2.4085322E-5
      }, {
        "text" : "world conflicts ii",
        "score" : 1.4253577E-5
      }, {
        "text" : "words conflicts ii",
        "score" : 4.8214292E-6
      }, {
        "text" : "wordy war ii",
        "score" : 4.1216194E-6
      } ]
    } ]
  }
}

As you can see, instead of the war term, the conflict term was returned for some of the phrase suggester results. So, our synonyms' configuration was taken into consideration. However, please remember that the synonyms were taken before the scoring of the fragments, so it can happen that the suggestions with the synonyms are not the ones that are scored the most, and you will not be able to see them in the suggester results.

The prefix suggester is based on the data structure called Finite State Transducer (FST) (http://en.wikipedia.org/wiki/Finite_state_transducer). Although it is highly efficient, it may require significant resources to build on systems with large amounts of data in them: systems that Elasticsearch is perfectly suitable for. If we would like to build such a structure on the nodes after each restart or cluster state change, we may lose performance. Because of this, the Elasticsearch creators decided to use an FST-like structure during index time and store it in the index so that it can be loaded into the memory when needed.

To use a prefix-based suggester we need to properly index our data with a dedicated field type called completion. It stores the FST-like structure in the index. In order to illustrate how to use this suggester, let's assume that we want to create an autocomplete feature to allow us to show book authors, which we store in an additional index. In addition to authors' names, we want to return the identifiers of the books they wrote in order to search for them with an additional query. We start with creating the authors index by running the following command:

curl -XPOST 'localhost:9200/authors' -d '{
 "mappings" : {
  "author" : {
   "properties" : {                
    "name" : { "type" : "string" },
    "ac" : {
     "type" : "completion",
     "index_analyzer" : "simple",
     "search_analyzer" : "simple",
     "payloads" : true
    }
   }
  }
 }
}'

Our index will contain a single type called author. Each document will have two fields: the name field, which is the name of the author, and the ac field, which is the field we will use for autocomplete. The ac field is the one we are interested in; we've defined it using the completion type, which will result in storing the FST-like structure in the index. In addition to this, we've used the simple analyzer for both index and query time. The last thing is payload, which is the additional information we will return along with the suggestion; in our case, it will be an array of book identifiers.

To index the data, we need to provide some additional information in addition to what we usually provide during indexing. Let's look at the following commands that index two documents describing authors:

curl -XPOST 'localhost:9200/authors/author/1' -d '{
 "name" : "Fyodor Dostoevsky",
 "ac" : {
  "input" : [ "fyodor", "dostoevsky" ],
  "output" : "Fyodor Dostoevsky",
  "payload" : { "books" : [ "123456", "123457" ] }
 }
}'
curl -XPOST 'localhost:9200/authors/author/2' -d '{
 "name" : "Joseph Conrad",
 "ac" : {
  "input" : [ "joseph", "conrad" ],
  "output" : "Joseph Conrad",
  "payload" : { "books" : [ "121211" ] }
 }
}'

Notice the structure of the data for the ac field. We provide the input, output, and payload properties. The payload property is used to provide additional information that will be returned. The input property is used to provide input information that will be used to build the FST-like structure and will be used to match the user input to decide whether the document should be returned by the suggester. The output property is used to tell the suggester which data should be returned for the document.

If the input and output property is the same in your case and you don't want to store payloads, you may index the documents just like you usually index your data. For example, the command to index our first document would look like this:

curl -XPOST 'localhost:9200/authors/author/3' -d '{
 "name" : "Stanislaw Lem",
 "ac" : [ "Stanislaw Lem" ]
}'

Finally, let's look at how to query our indexed data. If we would like to find documents that have authors starting with fyo, we would run the following command:

curl -XGET 'localhost:9200/authors/_suggest?pretty' -d '{
 "authorsAutocomplete" : {
  "text" : "fyo",
  "completion" : {
   "field" : "ac"
  }
 }
}'

Before we look at the results, let's discuss the query. As you can see, we've run the command to the _suggest endpoint, because we don't want to run a standard query; we are just interested in autocomplete results. The rest of the query is exactly the same as the standard suggester query run against the _suggest endpoint, with the query type set to completion.

The results returned by Elasticsearch for the preceding query look as follows:

{
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "authorsAutocomplete" : [ {
    "text" : "fyo",
    "offset" : 0,
    "length" : 3,
    "options" : [ {
      "text" : "Fyodor Dostoevsky",
      "score" : 1.0,
      "payload":{"books":["123456","123457"]}
    } ]
  } ]
}

As you can see, in response, we've got the document we were looking for along with the payload information, which is the identifier of the books for that author.

By default, the term frequency will be used to determine the weight of the document returned by the prefix suggester. However, this may not be the best solution when you have multiple shards for your index, or your index is composed of multiple segments. In such cases, it is useful to define the weight of the suggestion by specifying the weight property for the field defined as completion; the weight property should be set to a positive integer value and not a float one like the boost for queries and documents. The higher the weight property value, the more important the suggestion is. This gives us plenty of opportunities to control how the returned suggestions will be sorted.

For example, if we would like to specify a weight for the first document in our example, we would run the following command:

curl -XPOST 'localhost:9200/authors/author/1' -d '{
 "name" : "Fyodor Dostoevsky",
 "ac" : {
  "input" : [ "fyodor", "dostoevsky" ],
  "output" : "Fyodor Dostoevsky",
  "payload" : { "books" : [ "123456", "123457" ] },
  "weight" : 80
 }
}'

Now, if we would run our example query, the results would be as follows:

{
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "authorsAutocomplete" : [ {
    "text" : "fyo",
    "offset" : 0,
    "length" : 3,
    "options" : [ {
      "text" : "Fyodor Dostoevsky",
      "score" : 80.0,
      "payload":{"books":["123456","123457"]}
    } ]
  } ]
}

See how the score of the result changed. In our initial example, it was 1.0 and, now, it is 80.0; this is because we've set the weight parameter to 80 during the indexing.

There are three additional parameters supported by the suggester that we didn't mention till now. They are max_input_length, preserve_separators, and preserve_position_increments. Both preserve_separators and preserve_position_increments can be set to true or false. When setting the preserve_separators parameter to false, the suggester will omit separators such as whitespace (of course, proper analysis is required). Setting the preserve_position_increments parameter to false is needed if the first word in the suggestion is a stop word and we are using an analyzer that throws stop words away. For example, if we have The Clue as our document and the The word will be discarded by the analyzer by setting preserve_position_increments to false, the suggester will be able to return our document by specifying c as text.

The max_input_length property is set to 50 by default and specifies the maximum input length in UTF-16 characters. This limit is used at indexing time to limit the total number of characters stored in the internal structures.