Is NFS supported for OpenShift cluster internal components in Production?

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 3
  • 4
• NFS

Issue

• Is NFS Supported on Production for OpenShift internal components?
• Customers want to know if the NFS is going to be supported or not for production, and if it is not, the link to the documentation where it is mentioned.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

NFS V3 has not been tested in OpenShift 4 and should not be used (check the OpenShift Container Platform 4.x Tested Integrations for additional information). And while NFS V4 is supported as a storage backend in OpenShift, Red Hat strongly advises against using NFS to back OpenShift internal components in Production due to known issues.

IMPORTANT NOTE: The recommendation from Red Hat is that NFS should be used only with applications designed to work with the technical limitations of NFS. However, these NFS limitations have been found to have negative impacts on the core components for OpenShift, such as the integrated registry, the aggregate container logging infrastructure, the monitoring stack, and the cluster metrics and as such, should not be used with these whenever possible.

Integrated Image Registry

Based on the documentation (for This page is not included, but the link has been rewritten to point to the nearest parent document.OCP 3.11 and for OCP 4) it is supported to use NFS V4 as the storage provider, but there are known issues and it is not recommended:

• Specifically, if planning to use a scaled Image Registry, there are some issues and while Red Hat have outlined some ways to try and mitigate the impact, the only way to truly prevent the problems is to use a different storage provider.
• If permissions are set incorrectly in the NFS, pushing images to the Image Registry could fail as explained in Failed to push image with error filesystem: mkdir /registry/docker: permission denied in OpenShift 4.
• The error manifest blob unknown blob unknown to registry while pushing images to registry is an example of known issue, usually caused by corruption of an image or a layer.

OpenShift Logging

Loki

As explained in why Loki needs block and object storage in RHOCP 4?, Loki uses 2 different kind of storage. For the permanent storage, it must be S3 compatible Object storage, and for the temporary storage the recommended for best performance is Block storage. NFS is not recommended as it could cause performance issues and data corruption.

Elasticsearch

As explained in is NFS Supported for ElasticSearch Storage in OpenShift?, it is not supported to use NFS as for Elasticsearch storage.

Monitoring Stack

Based on the OpenShift 4 documentation for the This page is not included, but the link has been rewritten to point to the nearest parent document.persistent storage prerequisites for the monitoring stack, Prometheus does not support file systems that are non-POSIX compliant. Also, as per the Content from prometheus.io is not included.Prometheus upstream documentation, while NFS could be POSIX-compliant, most implementations are not.

For using a NFS file system for Prometheus storage, it is required to verify with the vendor that their NFS implementation is fully POSIX compliant. In addition to that, there are known issues when using NFS as storage for Prometheus like: Prometheus pods remain in CrashLoopBackOff and Prometheus pods in CrashLoopBackOff with error 'opening storage failed invalid block sequence'.
>Note: The storage for Prometheus needs to be configured as Block. Prometheus cannot use raw block volumes.

Cluster Metrics

Much like the integrated registry, based on the documentation (for OCP 3.11) NFS V4 is supported, but with explicit warn, and recommend, against the usage of NFS as it has been known to cause issues that are easily preventable by use of a different storage provider.

Root Cause

NFS V4 is supported as storage for OpenShift 4, but NFS V3 has not been tested in OpenShift 4. Even if NFS V4 is supported, the recommendation from Red Hat is that NFS should be used only with applications designed to work with the technical limitations of NFS. These NFS limitations have been found to have negative impacts on the core components for OpenShift.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.