As you already know (we've discussed it throughout Chapter 1, Introduction to Elasticsearch), every Elasticsearch index is built out of one or more shards and can have zero or more replicas. You also know that each of the shards and replicas are actual Apache Lucene indices that are built of multiple segments (at least one segment). If you recall, the segments are written once and read many times, and data structures, apart from the information about the deleted documents that are held in one of the files, can be changed. After some time, when certain conditions are met, the contents of some segments can be copied to a bigger segment, and the original segments are discarded and thus deleted from the disk. Such an operation is called segment merging.
You may ask yourself, why bother about segment merging? There are a few reasons. First of all, the more segments the index is built of, the slower the search will be and the more memory Lucene will need. In addition to this, segments are immutable, so the information is not deleted from it. If you happen to delete many documents from your index, until the merge happens, these documents are only marked as deleted and are not deleted physically. So, when segment merging happens, the documents that are marked as deleted are not written into the new segment, and this way, they are removed, which decreases the final segment size.
So, just to quickly summarize, segments merging takes place and from the user's point of view, it will result in two effects:
- It will reduce the number of segments in order to allow faster searching when a few segments are merged into a single one
- It will reduce the size of the index because of removing the deleted documents when the merge is finalized
However, you have to remember that segment merging comes with a price: the price of I/O operations, which can affect performance on slower systems. Because of this, Elasticsearch allows us to choose the merge policy and the store level throttling.
Although segment merging is Apache Lucene's duty, Elasticsearch allows us to configure which merge policy we would like to use. There are three policies that we are currently allowed to use:
tiered(the default one)log_byte_sizelog_doc
Each of the preceding mentioned policies have their own parameters, which define their behavior and the default values that we can override (please look at the section dedicated to the policy of your choice to see what those parameters are).
In order to tell Elasticsearch which merge policy we want to use, we should set index.merge.policy.type to the desired type, shown as follows:
index.merge.policy.type: tiered
Let's now look at the different merge policies and the functionality that they provide. After this, we will discuss all the configuration options provided by the policies.
The tiered merge policy is the default merge policy that Elasticsearch uses. It merges segments of approximately similar size, taking into account the maximum number of segments allowed per tier. It is also possible to differentiate the number of segments that are allowed to be merged at once from how many segments are allowed to be present per tier. During indexing, this merge policy will compute how many segments are allowed to be present in the index, which is called budget. If the number of segments the index is built of is higher than the computed budget, the tiered policy will first sort the segments by the decreasing order of their size (taking into account the deleted documents). After that, it will find the merge that has the lowest cost. The merge cost is calculated in a way that merges are reclaiming more deletions, and having a smaller size is favored.
If the merge produces a segment that is larger than the value specified by the index.merge.policy.max_merged_segment property, the policy will merge fewer segments to keep the segment size under the budget. This means that for indices that have large shards, the default value of the index.merge.policy.max_merged_segment property may be too low and will result in the creation of many segments, slowing down your queries. Depending on the volume of your data, you should monitor your segments and adjust the merge policy setting to match your needs.
The log byte size merge policy is a merge policy, which over time, will produce an index that will be built of a logarithmic size of indices. There will be a few large segments, then there will be a few merge factor smaller segments, and so on. You can imagine that there will be a few segments of the same level of size when the number of segments will be lower than the merge factor. When an extra segment is encountered, all the segments within that level are merged. The number of segments an index will contain is proportional to the logarithm of the next size in bytes. This merge policy is generally able to keep the low number of segments in your index while minimizing the cost of segments merging.
The log doc merge policy is similar to the log_byte_size merge policy, but instead of operating on the actual segment size in bytes, it operates on the number of documents in the index. This merge policy will perform well when the documents are similar in terms of size or if you want segments of similar sizes in terms of the number of documents.
We now know how merge policies work, but we lack the knowledge about the configuration options. So now, let's discuss each of the merge policies and see what options are exposed to us. Please remember that the default values will usually be OK for most of the deployments and they should be changed only when needed.
When using the tiered merge policy, the following options can be altered:
index.merge.policy.expunge_deletes_allowed: This defaults to10and specifies the percentage of deleted documents in a segment in order for it to be considered to be merged when runningexpungeDeletes.index.merge.policy.floor_segment: This is a property that enables us to prevent the frequent flushing of very small segments. Segments smaller than the size defined by this property are treated by the merge mechanism, as they would have the size equal to the value of this property. It defaults to2MB.index.merge.policy.max_merge_at_once: This specifies the maximum number of segments that will be merged at the same time during indexing. By default, it is set to10. Setting the value of this property to higher values can result in multiple segments being merged at once, which will need more I/O resources.index.merge.policy.max_merge_at_once_explicit: This specifies the maximum number of segments that will be merged at the same time during the optimize operation orexpungeDeletes. By default, this is set to30. This setting will not affect the maximum number of segments that will be merged during indexing.index.merge.policy.max_merged_segment: This defaults to5GBand specifies the maximum size of a single segment that will be produced during segment merging when indexing. This setting is an approximate value, because the merged segment size is calculated by summing the size of segments that are going to be merged minus the size of the deleted documents in these segments.index.merge.policy.segments_per_tier: This specifies the allowed number of segments per tier. Smaller values of this property result in less segments, which means more merging and lower indexing performance. It defaults to10and should be set to a value higher than or equal toindex.merge.policy.max_merge_at_once, or you'll be facing too many merges and performance issues.index.reclaim_deletes_weight: This defaults to2.0and specifies how many merges that reclaim deletes are favored. When setting this value to0.0, the reclaim deletes will not affect the merge selection. The higher the value, the more favored the merge that reclaims deletes will be.index.compund_format: This is a Boolean value that specifies whether the index should be stored in a compound format or not. It defaults tofalse. If set totrue, Lucene will store all the files that build the index in a single file. Sometimes, this is useful for systems running constantly out of file handlers, but it will decrease the searching and indexing performance.
When using the log_byte_size merge policy, the following options can be used to configure its behavior:
merge_factor: This specifies how often segments are merged during indexing. With a smallermerge_factorvalue, the searches are faster and less memory is used, but this comes with the cost of slower indexing. With largermerge_factorvalues, it is the opposite—the indexing is faster (because of less merging being done), but the searches are slower and more memory is used. By default,merge_factoris given the value of10. It is advised to use larger values ofmerge_factorfor batch indexing and lower values of this parameter for normal index maintenance.min_merge_size: This defines the size (the total size of the segment files in bytes) of the smallest segment possible. If a segment is lower in size than the number specified by this property, it will be merged if themerge_factorproperty allows us to do that. This property defaults to1.6MBand is very useful in order to avoid having many very small segments. However, one should remember that setting this property to a large value will increase the merging cost.max_merge_size: This defines the maximum size (the total size of the segment files in bytes) of the segment that can be merged with other segments. By default, it is not set, so there is no limit on the maximum size a segment can be in order to be merged.maxMergeDocs: This defines the maximum number of documents a segment can have in order to be merged with other segments. By default, it is not set, so there is no limit to the maximum number of documents a segment can have.calibrate_size_by_deletes: This is a Boolean value, which is set totrueand specifies whether the size of the deleted documents should be taken into consideration when calculating the segment size.
The mentioned properties we just saw should be prefixed with the index.merge.policy prefix. So if we would like to set the min_merge_docs property, we should use the index.merge.policy.min_merge_docs property.
In addition to this, the log_byte_size merge policy accepts the index.merge.async and index.merge.async_interval properties just like the tiered merge policy does.
When using the log_doc merge policy, the following options can be used to configure its behavior:
merge_factor: This is same as the property that is present in thelog_byte_sizemerge policy, so please refer to this policy for the explanation.min_merge_docs: This defines the minimum number of documents for the smallest segment. If a segment contains a lower document count than the number specified by this property, it will be merged if themerge_factorproperty allows this. This property defaults to1000and is very useful in order to avoid having many very small segments. However, one should remember that setting this property to a large value will increase the merging cost.max_merge_docs: This defines the maximum number of documents a segment can have in order to be merged with other segments. By default, it is not set, so there is no limit to the maximum number of documents a segment can have.calibrate_size_by_deletes: This is a Boolean value that defaults totrueand specifies whether the size of deleted documents should be taken into consideration when calculating the segment size.
Similar to the previous merge policy, the previously mentioned properties should be prefixed with the index.merge.policy prefix. So if we would like to set the min_merge_docs property, we should use the index.merge.policy.min_merge_docs property.
In addition to having control over how the merge policy is behaving, Elasticsearch allows us to define the execution of the merge policy once a merge is needed. There are two merge schedulers available, with the default being ConcurrentMergeScheduler.
This is a merge scheduler that will use multiple threads in order to perform segments' merging. This scheduler will create a new thread until the maximum number of threads is reached. If the maximum number of threads is reached and a new thread is needed (because segments' merge needs to be performed), all the indexing will be paused until at least one merge is completed.
In order to control the maximum threads allowed, we can alter the index.merge.scheduler.max_thread_count property. By default, it is set to the value calculated by the following equation:
maximum_value(1, minimum_value(3, available_processors / 2)
So, if our system has eight processors available, the maximum number of threads that the concurrent merge scheduler is allowed to use will be equal to four.
You should also remember that this is especially not good for spinning disks. You want to be sure that merging won't saturate your disks' throughput. Because of this, if you see extensive merging, you should lower the number of merging threads. It is usually said that for spinning disks, the number of threads used by the concurrent merge scheduler should be set to 1.
A simple merge scheduler uses the same thread for merging. It results in a merge that stops all the other document processing that was happening on the same thread, which in this case, means the stopping of indexing. This merge scheduler is only provided for backwards compatibility and, in fact, uses the concurrent merge scheduler with the number of threads equal to one.
In order to set the desired merge scheduler, one should set the index.merge.scheduler.type property to the value of concurrent or serial. For example, in order to use the concurrent merge scheduler, one should set the following property:
index.merge.scheduler.type: concurrent
In order to use the serial merge scheduler, one should set the following property:
index.merge.scheduler.type: serial
Note
When talking about the merge policy and merge schedulers, it would be nice to visualize them. If one needs to see how the merges are done in the underlying Apache Lucene library, we suggest that you visit Mike McCandless' blog post at http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html.
In addition to this, there is a plugin that allows us to see what is happening to the segments called SegmentSpy. Refer to the following URL for more information: