Understanding Elasticsearch memory usage in OpenShift

NOTE: This article refers to the Elasticsearch in OpenShift 3, different values are for when Elasticsearch is running on OpenShift 4 where the less recommended memory is 16G.

Elasticsearch appears to require a lot of memory. When looking at memory usage on a node running a logging-es pod, it can be noted that the ES process is using all the memory allocated to it in the INSTANCE_RAM environment variable and memory limit in the resources section of the logging-es DC. Even when adding more memory to the logging-es DC, it will consume all the new memory given to it. This article will briefly discuss this and recommendations on Elasticsearch sizing with regards to memory.

Elasticsearch is a very memory-bound application. That is to say, you'll see the most performance improvements with an increase in available memory for Elasticsearch; and Elasticsearch requires a lot of memory.

We recommend 8G allocated as a bare minimum. For most clusters we suggest at least 12G and for production clusters up to 16G or more. For most cases 16G will be enough, but in very busy clusters more may be required, such as 32G or the upper limit 64G. A good baseline when testing performance on a new cluster is to start with 12G and monitor performance. If logs take a particularly long time to load in the Kibana web console, or if fluentd fails to flush its buffers (which it will report in its log output), or if Elasticsearch hits out of memory issues, adding more memory is suggested. Another symptom of insufficient heap size is this error from the JVM for Elasticsearch: java.lang.OutOfMemoryError: Java heap space.

One other thing to note is that from the host's perspective, Elasticsearch will appear to be always consuming all the memory allocated to it. Elasticsearch will automatically claim an amount of memory equal to INSTANCE_RAM (defined in the logging-es dc) which it allocates to its heap and other Java processes. We suggest having INSTANCE_RAM be equal to the memory limit as listed in the resources section of the logging-es DC, and keeping the request blank or equal to the limit:

Limits:
  cpu:  2
  memory:   12G
Requests:
  cpu:  1
  memory:   12G

Note that this rule doesn't apply to CPU, which you can safely give a lower request than limit.

The above is also discussed in a related solution addressing a common error seen in Fluentd in undersized EFK stacks.

Elasticsearch memory allocation also has an effect on shard re-allocation. Upon startup, all shards in Elasticsearch will be in UNASSIGNED until they have been reassigned to an Elasticsearch node. This process will also be sped up with more memory.