The store module is one of the modules that we usually don't pay much attention to when configuring our cluster; however, it is very important. It is an abstraction between the I/O subsystem and Apache Lucene itself. All the operation that Lucene does with the hard disk drive is done using the store module. Most of the store types in Elasticsearch are mapped to an appropriate Apache Lucene Directory class (http://lucene.apache.org/core/4_9_0/core/org/apache/lucene/store/Directory.html). The directory is used to access all the files the index is built of, so it is crucial to properly configure it.
Elasticsearch exposes five store types that we can use. Let's see what they provide and how we can leverage their features.
The simplest implementation of the Directory class that is available is implemented using a random access file (Java RandomAccessFile—http://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html) and maps to SimpleFSDirectory (http://lucene.apache.org/core/4_9_0/core/org/apache/lucene/store/SimpleFSDirectory.html) in Apache Lucene. It is sufficient for very simple applications. However, the main bottleneck will be multithreaded access, which has poor performance. In the case of Elasticsearch, it is usually better to use the new I/O-based system store instead of the Simple filesystem store. However, if you would like to use this system store, you should set index.store.type to simplefs.
This store type uses the Directory class implementation based on the FileChannel class (http://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html) from java.nio package and maps to NIOFSDirectory in Apache Lucene (http://lucene.apache.org/core/4_9_0/core/org/apache/lucene/store/NIOFSDirectory.html). The discussed implementation allows multiple threads to access the same files concurrently without performance degradation. In order to use this store, one should set index.store.type to niofs.
Note
Please remember that because of some bugs that exist in the JVM machine for Microsoft Windows, it is very probable that the new I/O filesystem store will suffer from performance problems when running on Microsoft Windows. More information about this bug can be found at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734.
This store type uses Apache Lucene's MMapDirectory (http://lucene.apache.org/core/4_9_0/core/org/apache/lucene/store/MMapDirectory.html) implementation. It uses the mmap system call (http://en.wikipedia.org/wiki/Mmap) for reading, and it uses random access files for writing. It uses a portion of the available virtual memory address space in the process equal to the size of the file being mapped. It doesn't have any locking, so it is scalable when it comes to multithread access. When using mmap to read index files for the operating system, it looks like it is already cached (it was mapped to the virtual space). Because of this, when reading a file from the Apache Lucene index, this file doesn't need to be loaded into the operating system cache and thus, the access is faster. This basically allows Lucene and thus Elasticsearch to directly access the I/O cache, which should result in fast access to index files.
It is worth noting that the MMap filesystem store works best on 64-bit environments and should only be used on 32-bit machines when you are sure that the index is small enough and the virtual address space is sufficient. In order to use this store, one should set index.store.type to mmapfs.
Introduced in Elasticsearch 1.3.0, the hybrid file store uses both NIO and MMap access depending on the file type. A the time of writing this, only term dictionary and doc values were read and written using MMap, and all the other files of the index were opened using NIOFSDirectory. In order to use this store, one should set index.store.type to default.
This is the second store type that is not based on the Apache Lucene Directory (the first one is the hybrid filesystem store). The memory store allows us to store all the index files in the memory, so the files are not stored on the disk. This is crucial, because it means that the index data is not persistent—it will be removed whenever a full cluster restart will happen. However, if you need a small, very fast index that can have multiple shards and replicas and can be rebuilt very fast, the memory store type may be the thing you are looking for. In order to use this store, one should set index.store.type to memory.
When using the memory store type, we also have some degree of control over the caches, which are very important when using the memory store. Please remember that all the following settings are set per node:
cache.memory.direct: This defaults totrueand specifies whether the memory store should be allocated outside of the JVM heap memory. It is usually a good idea to leave it to the default value so that the heap is not overloaded with data.cache.memory.small_buffer_size: This defaults to1kband defines a small buffer size—the internal memory structure used to hold segments' information and deleted documents' information.cache.memory.large_buffer_size: This defaults to1mband defines a large buffer size—the internal memory structure used to hold index files other than segments' information and deleted documents.cache.memory.small_cache_size: The objects' small cache size—the internal memory structure used for the caching of index segments' information and deleted documents' information. It defaults to10mb.cache.memory.large_cache_size: The objects' large cache size—the internal memory structure used to cache information about the index other than the index segments' information and deleted documents' information. It defaults to500mb.
There are differences when it comes to the default store of Elasticsearch 1.3.0 and the newer and older versions.
Starting from Elasticsearch 1.3.0, the new default Elasticsearch store type is the hybrid one that we can choose by setting index.store.type to default.
By default, Elasticsearch versions older than 1.3.0 use filesystem-based storage. However different store types are chosen for different operating systems. For example, for a 32-bit Microsoft Windows system, the simplefs type will be used; mmapfs will be used when Elasticsearch is running on Solaris and Microsoft Windows 64 bit, and niofs will be used for the rest of the world.
Note
If you are looking for some information from experts on how they see which Directory implementation to use, please look at the http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html post written by Uwe Schindler and http://jprante.github.io/lessons/2012/07/26/Mmap-with-Lucene.html by Jörg Prante.
Usually, the default store type will be the one that you want to use. However, sometimes, it is worth considering using the MMap file system store type, especially when you have plenty of memory and your indices are big. This is because when using mmap to access the index file, it will cause the index files to be cached only once and be reused both by Apache Lucene and the operating system.