Elasticsearch Shard Rebalancing Tutorial

An Elasticsearch shard is a unit that allows the Elasticsearch engine to distribute data in a cluster. In Elasticsearch, we say that a cluster is “balanced” when it contains an equal number of shards on every node without having a large concentration of shards on a single node. Depending on how you configure Elasticsearch, it automatically performs shards rebalancing in your cluster to improve performance.

Automatic shard rebalancing conforms to restrictions and rules like allocation filtering and forced awareness, leading to the most efficient and well-balanced cluster possible.

NOTE: Do not confuse shard reallocation, which is the process of finding and moving unassigned shards to the nodes in which they reside, with rebalancing. Rebalancing takes assigned shards and moves them evenly to various nodes, the purpose being the equal distribution of shards per node.

How to Enable Automatic Rebalancing

To enable automatic cluster rebalancing in Elasticsearch, we can use the PUT request to_cluster API endpoint and add the settings we need.

The settings available for dynamic shard rebalancing include:

cluster.routing.rebalance.enable: Controls automatic rebalancing for various shard types, such as:
All: Sets enable shard rebalancing for all indices.
None: Disables shard rebalance for all indices.
Replicas: Only replica shard rebalance is allowed.
Primary: Only primary shard rebalancing is allowed.
cluster.routing.allocation.allow_rebalance: Sets the value for shard rebalancing. Options include:
- Always: Enables rebalancing indefinitely.
- Indices_primaries_active: Allows rebalancing only when all primary shards in the cluster are allocated.
- Indices_all_active: Allows rebalancing when only the shards in the cluster are allocated. This includes both the primary and the replica shards.

cluster.routing.allocation.cluster.concurrent.rebalance: This option sets the number of concurrent rebalances allowed in the cluster. The default value is 2.

Consider the request below to allow automatic shard rebalancing for the cluster.

PUT /_cluster/settings

"persistent":
"cluster.routing.rebalance.enable": "primaries",
"cluster.routing.allocation.allow_rebalance": "always" ,
"cluster.routing.allocation.cluster_concurrent_rebalance":"2"

The following is the cURL command:

curl -XPUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' "persistent": "cluster.routing.rebalance.enable": "primaries","cluster.routing.allocation.allow_rebalance": "always" ,"cluster.routing.allocation.cluster_concurrent_rebalance":"2" '

This command should return a response as the JSON object acknowledges the settings that are updated.

“acknowledged”: true,
"persistent" :
"cluster" :
"routing" :
"rebalance" :
"enable" : "primaries"
,
"allocation" :
"allow_rebalance" : "always",
"cluster_concurrent_rebalance" : "2"

,
"transient" :

Manual Index Rebalancing

You can also rebalance a shard manually for a specific index. I would not recommend this option because the Elasticsearch default rebalancing options are very efficient.

However, should the need to perform manual rebalancing arise, you can use the following request:

The cURL command is:

curl -XPOST "http://localhost:9200/_cluster/reroute" -H 'Content-Type: application/json' -d' "commands" : [ "move" : "index" : "kibana_sample_data_flights", "shard" : 0, "from_node" : "instance-0000000001", "to_node" : "instance-0000000002" ]'

NOTE: Keep in mind that if you perform a manual rebalance, Elasticsearch may move the shards automatically to ensure the best rebalance possible.

Conclusion

This guide walked you through updating and modifying the settings for an Elasticsearch cluster to enable automatic shard rebalancing. The article also covered manual rebalancing, if you require it.