Failed to deploy Elastic Search due to "Unable to load index mapping for io.fabric8.elasticsearch.kibana.mapping.empty. The key was not in the settings or it specified a file that does not exists."

Environment

• Red Hat OpenShift Container Platform 3.5.5.15
• Kubernetes v1.5.2+43a9be4
• etcd 3.1.0

Issue

• Elastic search re-deployment failed after cleaning up storage
• The following error is present in the log file:

[INFO ][container.run            ] ES_JAVA_OPTS: '-Dmapper.allow_dots_in_name=true -Xms4096m -Xmx4096m'
Exception in thread "main" java.lang.RuntimeException: Unable to load index mapping for io.fabric8.elasticsearch.kibana.mapping.empty.  The key was not in the settings or it specified a file that does not exists.
	at io.fabric8.elasticsearch.plugin.kibana.IndexMappingLoader.loadMapping(IndexMappingLoader.java:56)
	at io.fabric8.elasticsearch.plugin.kibana.IndexMappingLoader.<init>(IndexMappingLoader.java:42)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at <<<guice>>>
	at org.elasticsearch.node.Node.<init>(Node.java:213)
	at org.elasticsearch.node.Node.<init>(Node.java:140)
	at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:45)

Resolution

This is a known issue in OpenShift Container Platform 3.5 version and can be resolved by applying RHBA-2017:1810 .

There is a work around option if unable to apply the errata above. The steps are provided below:

• Login to master:

$oc login ..

• Go to the logging project:

$oc project logging

• Edit the elastic search configuration:

$ oc edit configmap logging-elasticsearch

• Add the index io.fabric8.elasticsearch.kibana.mapping.empty after the other indices in the configmap:

...
    io.fabric8.elasticsearch.authentication.users: ["system.logging.kibana", "system.logging.fluentd", "system.logging.curator", "system.admin"]
    io.fabric8.elasticsearch.kibana.mapping.app: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json
    io.fabric8.elasticsearch.kibana.mapping.ops: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json
    io.fabric8.elasticsearch.kibana.mapping.empty: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json
...

• Save and close the configmap
• Scale down all the ES instances

$ for dc in `oc get dc -o name --selector=component=es`; do oc scale $dc --replicas=0; done

• After all the logging-es pods have been terminated, scale up all the ES instances so that they can get the latest configuration

$ for dc in `oc get dc -o name --selector=component=es`; do oc scale $dc --replicas=1; done

Root Cause

• io.fabric8.elasticsearch.kibana.mapping.empty property is missing from configuration file leading to elastic search failing to start and generating a stack trace.

Diagnostic Steps

• Please provide the log for the elastic search logging pod:

$ oc get po -n logging --selector=component=es -o 'go-template={{range $pod := .items}}oc logs {{$pod.metadata.name}} > {{$pod.metadata.name}}.log{{"\n"}}{{end}}' | bash -

• Provide also the configmap

$ oc get configmap logging-elasticsearch -o yaml > logging-elasticsearch-configmap.yaml

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.