Planning to run containerized DB or nosql workloads on Openshift Container Storage ?

Solution Verified - Updated

Environment

  • Openshift Container Storage - 3.9/3.10/3.11
  • Red Hat Gluster Storage - 3.X

Issue

  • The pgbench perf benchmark for PGSQL does not complete
  • Issue with open-behind and write-behind xlators in Red Hat Gluster Storage when gluster-file is being used for containerized DB workload.
  • Database files are reported corrupted when stored on Red Hat Gluster Storage.

Resolution

  • Persistent volumes backed by block storage is the recommended method for the following workloads:
    Jenkins
    ElasticSearch
    Prometheus

  • When using glusterfs (file based storage) for transactional workloads , adapt performance translators per below recommendation in Red Hat Gluster Storage / OCS and follow the steps below :

Note : In case of OCS, below commands should be be executed inside one OCS pod 

 # gluster volume set VOLUME performance.strict-o-direct on
 # gluster volume set VOLUME performance.open-behind off
 # gluster volume set VOLUME performance.write-behind off
 # gluster volume set VOLUME performance.stat-prefetch off
 # gluster volume set VOLUME performance.quick-read off
 # gluster volume set VOLUME performance.read-ahead off
 # gluster volume set VOLUME performance.io-cache off
 # gluster volume set VOLUME performance.readdir-ahead off
  • In order to verify the existing status run command below on gluster node, in case of OCS execute below commands inside one OCS pod 
 # gluster volume get VOLUME all  | grep <performance translator>

for example 

 # gluster volume get VOLUME all  | egrep "performance.stat-prefetch |performance.write-behind |performance.open-behind |performance.quick-read |performance.strict-o-direct |performance.read-ahead |performance.io-cache |performance.readdir-ahead "

For cases when dynamic storage provisioning is used, then it is possible to create storage class with desiredvolumeoptions in parameters section of storage class - an example what to specify in storage class definition is showed below 

parameters:
  resturl: http://heketi-storage-glusterfs.router.default.svc.cluster.local
  restuser: admin
  secretName: heketi-storage-admin-secret
  secretNamespace: glusterfs
  volumeoptions: performance.strict-o-direct on performance.stat-prefetch off performance.write-behind off performance.open-behind off performance.quick-read off performance.read-ahead off performance.io-cache off performance.readdir-ahead off

The storage class defined with desired volumeoptions will ensure that PVC created based on that storage class will always get correct volume options and possibility of mistakes is smaller than for case when manually setting these options.

NOTE. Above steps are now included in OCS 3.11 Deployment Documentation , Chapter 1 :
If using file storage for transactional workloads, turn off the performance translators as described in Chapter 11, Setting up Custom Volume Options

Root Cause

  • For more information, please contact Red Hat Support
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.