Query templates

When the application grows, it is very probable that the environment will start to be more and more complicated. In your organization, you probably have developers who specialize in particular layers of the application—for example, you have at least one frontend designer and an engineer responsible for the database layer. It is very convenient to have the development divided into several modules because you can work on different parts of the application in parallel without the need of constant synchronization between individuals and the whole team. Of course, the book you are currently reading is not a book about project management, but search, so let's stick to that topic. In general, it would be useful, at least sometimes, to be able to extract all queries generated by the application, give them to a search engineer, and let him/her optimize them, in terms of both performance and relevance. In such a case, the application developers would only have to pass the query itself to Elasticsearch and not care about the structure, query DSL, filtering, and so on.

Introducing query templates

With the release of Elasticsearch 1.1.0, we were given the possibility of defining a template. Let's get back to our example library e-commerce store that we started working on in the beginning of this book. Let's assume that we already know what type of queries should be sent to Elasticsearch, but the query structure is not final—we will still work on the queries and improve them. By using the query templates, we can quickly supply the basic version of the query, let application specify the parameters, and modify the query on the Elasticsearch side until the query parameters change.

Let's assume that one of our queries needs to return the most relevant books from our library index. We also allow users to choose whether they are interested in books that are available or the ones that are not available. In such a case, we will need to provide two parameters—the phrase itself and the Boolean that specifies the availability. The first, simplified example of our query could looks as follows:

{
 "query": {
  "filtered": {
   "query": {
    "match": {
     "_all": "QUERY"
    }
   },
   "filter": {
    "term": {
     "available": BOOLEAN
    }
   }
  }
 }
}

The QUERY and BOOLEAN are placeholders for variables that will be passed to the query by the application. Of course, this query is too simple for our use case, but as we already said, this is only its first version—we will improve it in just a second.

Having our first query, we can now create our first template. Let's change our query a bit so that it looks as follows:

{
  "template": {
   "query": {
    "filtered": {
     "query": {
      "match": {
       "_all": "{{phrase}}"
      }
     },
     "filter": {
      "term": {
       "available": "{{avail}}"
      }
     }
    }
   }
  },
  "params": {
   "phrase": "front",
   "avail": true
  }
}

You can see that our placeholders were replaced by {{phrase}} and {{avail}}, and a new section params was introduced. When encountering a section like {{phrase}}, Elasticsearch will go to the params section and look for a parameter called phrase and use it. In general, we've moved the parameter values to the params section, and in the query itself we use references using the {{var}} notation, where var is the name of the parameter from the params section. In addition, the query itself is nested in the template element. This way we can parameterize our queries.

Let's now send the preceding query to the /library/_search/template REST endpoint (not the /library/_search as we usually do) using the GET HTTP method. To do this, we will use the following command:

curl -XGET 'localhost:9200/library/_search/template?pretty' -d '{
 "template": {
   "query": {
    "filtered": {
     "query": {
      "match": {
       "_all": "{{phrase}}"
      }
     },
     "filter": {
      "term": {
       "available": "{{avail}}"
      }
     }
    }
   }
  },
  "params": {
   "phrase": "front",
   "avail": true
  }
}'

Templates as strings

The template can also be provided as a string value. In such a case, our template will look like the following:

{
  "template": "{ "query": { "filtered": { "query": {  "match": { "_all": "{{phrase}}" } }, "filter": {  "term": { "available": "{{avail}}" } } } } }",
   "params": {
      "phrase": "front",
      "avail": true
   }
}

As you can see, this is not very readable or comfortable to write—every quotation needs to be escaped, and new line characters are also problematic and should be avoided. However, you'll be forced to use this notation (at least in Elasticsearch from 1.1.0 to 1.4.0 inclusive) when you want to use Mustache (a template engine we will talk about in the next section) features.

Note

There is a gotcha in the Elasticsearch version used during the writing of this book. If you prepare an incorrect template, the engine detects an error and writes info into the server logs, but from the API point of view, the query is silently ignored and all documents are returned, just like you would send the match_all query. You should remember to double-check your template queries until that is changed.

The Mustache template engine

Elasticsearch uses Mustache templates (see: http://mustache.github.io/) to generate resulting queries from templates. As you have already seen, every variable is surrounded by double curly brackets and this is specific to Mustache and is a method of dereferencing variables in this template engine. The full syntax of the Mustache template engine is beyond the scope of this book, but we would like to briefly introduce you to the most interesting parts of it: conditional expression, loops, and default values.

Note

The detailed information about Mustache syntax can be found at http://mustache.github.io/mustache.5.html.

Conditional expressions

The {{val}} expression results in inserting the value of the val variable. The {{#val}} and {{/val}} expressions inserts the values placed between them if the variable called val computes to true.

Let's take a look at the following example:

curl -XGET 'localhost:9200/library/_search/template?pretty' -d '{
  "template": "{ {{#limit}}"size": 2 {{/limit}}}",
  "params": {
   "limit": false
  }
}'

The preceding command returns all documents indexed in the library index. However, if we change the limit parameter to true and send the query once again, we would only get two documents. That's because the conditional would be true and the template would be activated.

Note

Unfortunately, it seems that versions of Elasticsearch available during the writing of this book have problems with conditional expressions inside templates. For example, one of the issues related to that is available at https://github.com/elasticsearch/elasticsearch/issues/8308. We decided to leave the section about conditional expressions with the hope that the issues will be resolved soon. The query templates can be a very handy functionality when used with conditional expressions.

Loops

Loops are defined between exactly the same as conditionals—between expression {{#val}} and {{/val}}. If the variable from the expression is an array, you can insert current values using the {{.}} expression.

For example, if we would like the template engine to iterate through an array of terms and create a terms query using them, we could run a query using the following command:

curl -XGET 'localhost:9200/library/_search/template?pretty' -d '{
  "template": {
   "query": {
    "terms": {
     "title": [
      "{{#title}}",
      "{{.}}",
      "{{/title}}"
     ]
    }
   }
  },
  "params": {
   "title": [ "front", "crime" ]
  }
}'

Default values

The default value tag allows us to define what value (or whole part of the template) should be used if the given parameter is not defined. The syntax for defining the default value for a variable called var is as follows:

{{var}}{{^var}}default value{{/var}}

For example, if we would like to have the default value of crime for the phrase parameter in our template query, we could send a query using the following command:

curl -XGET 'localhost:9200/library/_search/template?pretty' -d '{
  "template": {
   "query": {
    "term": {
     "title": "{{phrase}}{{^phrase}}crime{{/phrase}}"
    }
   }
  },
  "params": {
   "phrase": "front"
  }
}'

The preceding command will result in Elasticsearch finding all documents with term front in the title field. However, if the phrase parameter was not defined in the params section, the term crime will be used instead.

Storing templates in files

Regardless of the way we defined our templates previously, we were still a long way from decoupling them from the application. We still needed to store the whole query in the application, we were only able to parameterize the query. Fortunately, there is a simple way to change the query definition so it can be read dynamically by Elasticsearch from the config/scripts directory.

For example, let's create a file called bookList.mustache (in the config/scripts/ directory) with the following contents:

{
 "query": {
  "filtered": {
   "query": {
    "match": {
     "_all": "{{phrase}}"
    }
   },
   "filter": {
    "term": {
     "available": "{{avail}}"
    }
   }
  }
 }
}

We can now use the contents of that file in a query by specifying the template name (the name of the template is the name of the file without the .mustache extension). For example, if we would like to use our bookList template, we would send the following command:

curl -XGET 'localhost:9200/library/_search/template?pretty' -d '{
  "template": "bookList",
  "params": {
   "phrase": "front",
   "avail": true
  }
}'

Note

The very convenient fact is that Elasticsearch can see the changes in the file without the need of a node restart. Of course, we still need to have the template file stored on all Elasticsearch nodes that are capable of handling the query execution. Starting from Elasticsearch 1.4.0, you can also store templates in a special index called .scripts. For more information please refer to the official Elasticsearch documentation available at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-template.html.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset