Elasticsearch monitoring APIs expose a lot of information, both about the search engine itself as well as about the environment, such as the operating system. We saw that in Chapter 10, Administrating Your Cluster. Because of this and the ease of retrieving this information, numerous applications were built – ones that allow us to do monitoring and beyond. Some of these applications are simple and just read the data in real time without any persistent storage, while others allow us to read historical data about our cluster behavior. In this chapter, we will only slightly touch the top of the pile of information about such applications, but we strongly advise you to get familiar with some of them as they can make your everyday work with Elasticsearch easier.
We chose three examples of monitoring solutions which take a different approach of integration with Elasticsearch. The first two tools are available as Elasticsearch plugins and the third takes a different approach to integration.
This tool is available as an Elasticsearch plugin but can also be downloaded separately as a JavaScript application run in a browser.
Elasticsearch HQ uses JavaScript and AJAX techniques where data is fetched periodically from the cluster, prepared for visualization on the browser side, and shown to the user.
The tool allows us to track statistics on a particular node. The browser can present vital information about the cluster and particular nodes. The following screenshot shows the graphical user interface from Elasticsearch HQ:
We have the basic information about the cluster, the number of nodes, and Elasticsearch health. We can also see which node we are looking at and some statistics about the node, which include the memory usage (both heap and non-heap), the number of threads, Java virtual machine garbage collector work, and so on. The plugin also presents simplified information about schema and shards and allows execution of simple queries.
In order to install Elasticsearch HQ, one should just run the following command:
bin/plugin install royrusso/elasticsearch-HQAfter that, the GUI will be available at http://localhost:9200/_plugin/hq/.
One thing to remember is that Elasticsearch HQ doesn't persist the fetched data anywhere, so the data is only fetched when your browser is running and has Elasticsearch HQ opened. If something has happened in the past, you won't be able to diagnose it.
Marvel is the tool created by the Elasticsearch team. In the current version, it is built as a plugin for a visualization platform called Kibana (https://www.elastic.co/products/kibana).
Marvel also visualizes basic information about clusters and nodes by drawing nice graphs that are dynamically updated over time. The main difference from Elasticsearch HQ is that the performance data is stored on the server side (in the same or external Elasticsearch cluster), so historical data is available. The example screenshot is presented next:
The installation procedure for Marvel contains three steps:
bin/plugin install license bin/plugin install marvel-agent
And finally, the third step is to install the Marvel plugin in Kibana by running the following command:
bin/kibana plugin --install elasticsearch/marvel/latestThis tool presents a different approach than the previously mentioned tools. SPM is a Software as a Service (SaaS) solution created for monitoring Elasticsearch installations of any size and allows monitoring several clusters and different technologies. Though its roots are SaaS-based, it is also available on premises, which means that you can run SPM on your own machines without the need for sending your metrics to cloud.
Information is sent by simple client software installed on the Elasticsearch machine to the SPM servers. The main advantage is the possibility of storing information for a wider range of time and seeing what was happening in the past. You can create your own dashboards and correlate metrics with logs between multiple applications (SPM allows you to monitor a wide variety of applications).
The following screenshot shows the dashboard of SPM for Elasticsearch:
The overview dashboard shown in the preceding screenshot provides information about the cluster nodes, the request rate and latency, the number of documents in the indices, CPU usage, load, memory details, Java virtual machine memory, the disk space usage, and finally network traffic. You can get detailed information about each of these elements by going into the tab dedicated to it.
You can find additional information about SPM installation and available options at http://sematext.com/spm/index.html.