Need to manually rebalance or reallocate primary shards on Elasticsearch nodes

Environment

• Red Hat OpenShift Container Platform
  • 3.9

Issue

• We hit the low-disk watermark on an ES pod, or storage is getting full
• Getting "replicas will not be assigned to this node" log

[2019-03-13 21:54:41,351][INFO ][cluster.routing.allocation.decider] [logging-es-24mgtsne] low disk watermark [85%] exceeded on [xxxxxxxxxxxxxxx][logging-es-xxxxxxxx][/elasticsearch/persistent/logging-es/data/logging-es/nodes/0] free: 193.2gb[14.7%], replicas will not be assigned to this node

• How to reroute shards in CLUSTER_RECOVERED state

Resolution

It's suggested to have multiple primary shards, so that the logs are split among all the ES nodes. See this related solution for more.

It's possible to manually reallocate shards to other ES nodes.

• Get the node names of the logging-es nodes

# es_pod=NAME-OF-ES-POD
# oc exec $es_pod -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca https://localhost:9200/_cat/nodes?v

• Choose a shard to reallocate.

$ curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca https://localhost:9200/_cat/shards?v
index                                                                        shard prirep state    docs   store ip          node                            
....
project.ruby.38f2d6a2-46a1-11e9-b69f-fa163e1a1d73.2019.03.14                 0     p      STARTED    59 101.4kb 10.130.0.9  logging-es-data-master-jrjey15a 
....

• Reallocate it. Place the index name in the index field, choose the shard as noted in the previous command, and choose a node that does not already have a primary shard for this index.

$ curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca -XPOST 'https://localhost:9200/_cluster/reroute' -d '{
    "commands" : [ {
        "move" :
            {
              "index" : "project.ruby.38f2d6a2-46a1-11e9-b69f-fa163e1a1d73.2019.03.14", "shard" : 0,
              "from_node" : "logging-es-data-master-jrjey15a", "to_node" : "logging-es-data-master-9u3pliyt"             }
        }
]
}'

• Verify it moved

$ curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca https://localhost:9200/_cat/shards?v
index                                                                        shard prirep state    docs   store ip          node                            
....
project.ruby.38f2d6a2-46a1-11e9-b69f-fa163e1a1d73.2019.03.14                 0     p      STARTED    59 101.4kb 10.128.0.10 logging-es-data-master-9u3pliyt 
....

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.