Suggested NFS mount optimizations with NetApp Trident storage on OpenShift

Solution Verified - Updated 12 Aug 2026

Environment

OpenShift Container Platform 4.x
NetApp Trident CSI driver
NFS-based StorageClass (NFSv4.1+)
Red Hat Enterprise Linux CoreOS (RHCOS) worker and control-plane nodes
NetApp ONTAP storage backend
OpenShift Virtualization 4.x

Issue

How can I improve performance for persistent volumes provided by the ONTAP Trident CSI driver using NFS storage backed by ONTAP?

Resolution

Note: The parameters in this procedure are baseline recommendations, not definitive configurations for all environments. Storage performance is highly dependent on workload characteristics, network topology, and backend storage configuration. Customers should work with NetApp support to determine optimal settings for their specific deployment. These settings have been known to alleviate latency issues in some environments, but further tuning may be required.

This procedure tunes NFS client parameters on OpenShift nodes, Trident backend mount options, and ONTAP server-side session slots to reduce latency and improve concurrent I/O handling.

Steps 1 and 2 should be applied as a baseline and performance re-evaluated.
If VMs are still underperforming then use Step 3 to incrementally increase concurrency. Each time you iterate you will need to unmount/remount the pvcs via start/stop or live migration.

Prerequisites

If desired, enable the node-exporter mountstats Collector in OpenShift to query NFS metrics before starting the procedure.

Please verify the settings below are completed:

Cluster-admin (oc) access.
A maintenance window. Each pool cordons, drains, and reboots its nodes serially (one at a time by default). The master pool rollout cycles etcd members and API servers one node at a time — safe in principle (it's the same mechanism as control-plane OS updates) but it is a control-plane operation, so treat it accordingly.
Confirm PodDisruptionBudgets won't block drains, or the rollout stalls.

Never roll both pools at once — finish one pool before starting the other so you aren't draining control plane and workers concurrently.

Step 1: Update Trident Backend Configuration

Change the Trident configuration to modify the mount options in either the backend or the storage class:

kind: TridentBackendConfig  
spec:  
    nfsMountOptions: sec=sys,nconnect=8

StorageClass example:

mountOptions:
   - sec=sys
   - nconnect=8

NOTE: The StorageClass will need to be removed and recreated with these settings. Make sure to use the same StorageClass name for continuity.

After making this change, nodes need to be rebooted in order to flush existing settings and allow for all new mounts to use the updated nconnect setting.

Live Migrate or Start/Stop VMs

Modifying the trident backend settings requires that virtual machines either undergo a live migration or be restarted (start/stop) following the modification of the tridentbackendconfig and a node reboot. This process ensures the persistent volumes (PVs) are remounted with the new configuration; no further changes to the PVs or the virtual machines themselves are necessary.

Step 2: NFS Client Tuning on OpenShift — MachineConfig (worker + master)

Tune NFSv4.1 client kernel parameters (max_session_slots, sunrpc.tcp_max_slot_table_entries) on worker and control-plane (master) nodes via two parallel MachineConfigs, with rollout, validation, and rollback.

What this sets

A oneshot systemd unit (nfs-slot-tuning.service) that, at boot:

loads the sunrpc and nfs modules (so their /sys/module/.../parameters/ paths exist), then
writes the live values:

/sys/module/sunrpc/parameters/tcp_max_slot_table_entries = 128
/sys/module/nfs/parameters/max_session_slots             = 1024

Parameter	Layer	Notes
`nfs max_session_slots=1024`	NFSv4.1 session fore-channel slots	`1024` is the kernel hard ceiling (`NFS4_MAX_SLOT_TABLE`). The client requests this; the server negotiates down (ONTAP commonly grants ~180). Only applies to NFSv4.1+ sessions — if mounts negotiate to v4.0/v3 it has no effect.
`sunrpc tcp_max_slot_table_entries=128`	RPC transport slots (per TCP connection)	Modern kernel default ceiling is 65536 with dynamic growth, so `128` sets an explicit lower cap — not an increase. Harmless (128 RPCs/transport far exceeds need); don't describe it as "raising" the slot table.

Why a systemd unit, not modprobe.d or Tuned alone: the unit loads the modules (so the paths exist even on an idle node) and writes the live parameter (so it works on already-loaded modules with no reload). The value lives in exactly one place (the unit), and success/failure is visible via systemctl status — unlike Tuned's [sysfs] write, which fails silently if a path is missing.

Prerequisites

Cluster-admin (oc) access.
A maintenance window. Each pool cordons, drains, and reboots its nodes serially (one at a time by default). The master pool rollout cycles etcd members and API servers one node at a time — safe in principle (it's the same mechanism as control-plane OS updates) but it is a control-plane operation, so treat it accordingly.
Confirm PodDisruptionBudgets won't block drains, or the rollout stalls.
Never roll both pools at once — finish one pool before starting the other so you aren't draining control plane and workers concurrently.

Apply — worker

# mc-nfs-tuning-worker.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-nfs-tuning
spec:
  config:
    ignition:
      version: 3.4.0
    systemd:
      units:
        - name: nfs-slot-tuning.service
          enabled: true
          contents: |
            [Unit]
            Description=NFS client slot-table tuning (sunrpc + nfs module params)
            After=systemd-modules-load.service
            DefaultDependencies=no

            [Service]
            Type=oneshot
            RemainAfterExit=yes
            # load the modules so the sysfs paths exist (no-op if already loaded)
            ExecStartPre=/usr/sbin/modprobe sunrpc
            ExecStartPre=/usr/sbin/modprobe nfs
            # write the LIVE values (runtime-writable; no module reload needed)
            ExecStart=/usr/bin/sh -c 'echo 128 > /sys/module/sunrpc/parameters/tcp_max_slot_table_entries'
            ExecStart=/usr/bin/sh -c 'echo 1024 > /sys/module/nfs/parameters/max_session_slots'

            [Install]
            WantedBy=multi-user.target

oc apply -f mc-nfs-tuning-worker.yaml
# wait for the worker pool to fully complete (see Monitor) BEFORE applying master

Apply — master

Identical unit; only the role label and name differ. Apply after worker has finished.

# mc-nfs-tuning-master.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-master-nfs-tuning
spec:
  config:
    ignition:
      version: 3.4.0
    systemd:
      units:
        - name: nfs-slot-tuning.service
          enabled: true
          contents: |
            [Unit]
            Description=NFS client slot-table tuning (sunrpc + nfs module params)
            After=systemd-modules-load.service
            DefaultDependencies=no

            [Service]
            Type=oneshot
            RemainAfterExit=yes
            ExecStartPre=/usr/sbin/modprobe sunrpc
            ExecStartPre=/usr/sbin/modprobe nfs
            ExecStart=/usr/bin/sh -c 'echo 128 > /sys/module/sunrpc/parameters/tcp_max_slot_table_entries'
            ExecStart=/usr/bin/sh -c 'echo 1024 > /sys/module/nfs/parameters/max_session_slots'

            [Install]
            WantedBy=multi-user.target

oc apply -f mc-nfs-tuning-master.yaml

Changing the values

Edit the two echo numbers in the unit and re-apply — values are inline and human-readable (no base64). A value change re-renders the pool config and triggers another rolling reboot.

Monitor the rollout

Per pool (run for worker, then master):

oc get mcp worker -w
oc get mcp master -w
# Done when: UPDATING=False, UPDATED=True, DEGRADED=False, UPDATEDMACHINECOUNT == MACHINECOUNT

Per-node progress (swap the label for master). The MCO updates desiredConfig one node at a time, as it selects each node to work — it does not stamp the new config onto every node up front. So a node that hasn't started yet still shows currentConfig == desiredConfig (both on the old config) and state Done, which looks identical to a finished node. You therefore can't judge progress from current vs desired alone — you must compare each node's currentConfig against the target rendered config the pool is rolling toward:

# the rendered config the pool is targeting:
TARGET=$(oc get mcp worker -o jsonpath='{.spec.configuration.name}')
echo "target: $TARGET"

# per node: current vs desired vs state — compare CURRENT to $TARGET
oc get nodes -l node-role.kubernetes.io/worker -o custom-columns=\
'NODE:.metadata.name,'\
'STATE:.metadata.annotations.machineconfiguration\.openshift\.io/state,'\
'CURRENT:.metadata.annotations.machineconfiguration\.openshift\.io/currentConfig,'\
'DESIRED:.metadata.annotations.machineconfiguration\.openshift\.io/desiredConfig'

Read it against $TARGET:

CURRENT == $TARGET, state Done → finished (on the new config).
DESIRED == $TARGET but CURRENT still the old hash, state Working → updating now (the single in-flight node — draining/rebooting). desired only flips to the target when the MCO picks this node.
CURRENT == DESIRED but both still the old hash, state Done → not started yet (looks "done," but it's on the old config — the tell is the hash, not the equality).

So progress = how many nodes' CURRENT matches $TARGET, advancing one at a time — not the current/desired equality.

If a pool stalls (DEGRADED=True, or a Working node makes no progress for 10+ min — usually a blocked drain):

oc get pods -n openshift-machine-config-operator -o wide | grep <working-node>
oc logs -n openshift-machine-config-operator <mcd-pod> -c machine-config-daemon -f
# look for: "error when evicting pod ..." (PodDisruptionBudget / pod with no controller)

Validate on a node

After a node's currentConfig matches the pool target ($TARGET above) with state Done:

# (a) the unit ran cleanly
oc debug node/<node> -- chroot /host systemctl status nfs-slot-tuning.service --no-pager

# (b) live kernel values
oc debug node/<node> -- chroot /host sh -c \
  'cat /sys/module/sunrpc/parameters/tcp_max_slot_table_entries; \
   cat /sys/module/nfs/parameters/max_session_slots'
# expect: 128  and  1024

Because the unit force-loads the modules, this works even on a node not using NFS — both paths exist and both values are set after boot. (This is the key advantage over the modprobe.d approach, which left /sys/module/nfs/... absent on idle nodes.)

Sweep both pools:

for role in worker master; do
  echo "##### $role #####"
  for n in $(oc get nodes -l node-role.kubernetes.io/$role -o name); do
    echo "== $n =="
    oc debug $n -- chroot /host sh -c \
      'cat /sys/module/sunrpc/parameters/tcp_max_slot_table_entries; \
       cat /sys/module/nfs/parametersmax_session_slots' 2>/dev/null
  done
done

Troubleshooting: if a value is wrong, check the unit first — systemctl status nfs-slot-tuning.service and journalctl -u nfs-slot-tuning.service. A failed unit or missing module (lsmod | grep -E '^nfs|^sunrpc') means the modprobe/echo didn't run.

Step 3: ONTAP NFSv4.1+ session slots: Incremental tuning guide

The ONTAP SVM option -v4.x-session-num-slots controls the size of the NFSv4.1 and NFSv4.2 session slot table. Both the server and client advertise this maximum during CREATE_SESSION; the session negotiates down to min(client-offer, server-offer). Effective concurrency is bounded by whichever side is lower.

Default: 180 slots per session
Range: 1–2000
Privilege required: advanced
Scope: per SVM (not cluster-wide)

TIP: Raise the client-side max_session_slots to 1024 in one step. The incremental approach taken on the ONTAP side will effectively limit the session slots available without having to do the more disruptive client side equivalent.

Command syntax

# Enter advanced privilege
set -privilege advanced

# Show current value
vserver nfs show -vserver <svm_name> -fields v4.x-session-num-slots

# Set value
vserver nfs modify -vserver <svm_name> -v4.x-session-num-slots <value>

# Return to admin privilege
set -privilege admin

IMPORTANT: The change takes effect for new sessions only. Existing NFSv4.x sessions retain their negotiated slot count until they are torn down and re-established (client remount or session expiry).

Incremental rollout plan

The goal is to validate at each step before proceeding. Start with the ONTAP default of 180 and only increase when it can be safely determined that ONTAP resources aren’t being overconsumed.

Step	Value	ONTAP Command
0	180	Keep default; enable monitoring
1	256	`vserver nfs modify -vserver <svm> -v4.x-session-num-slots 256`
2	512	`vserver nfs modify -vserver <svm> -v4.x-session-num-slots 512`
3	768	`vserver nfs modify -vserver <svm> -v4.x-session-num-slots 768`
4	1024	`vserver nfs modify -vserver <svm> -v4.x-session-num-slots 1024`

+256 cadence rationale: Slot tables consume server memory per session and allow more concurrency from a single NFS session. Incremental changes isolate the benefit at each tier and limit blast radius if a value causes unexpected behavior.

Rollback

Step 1: Revert Trident Backend Configuration

Change the ARGOCD Trident backend configuration to modify the mount options in the backend:

kind: TridentBackendConfig  
spec:  
nfsMountOptions: sec=sys

This will not affect running pods or VMs until they are restarted in such a way where the mounts of the volumes for those pods or VMS are remounted.

Step 2: Revert ONTAP NFSv4.1+ session slots

If a step produces latency regression or unexpected behavior, revert immediately:

set -privilege advanced
vserver nfs modify -vserver <svm_name> -v4.x-session-num-slots 180
set -privilege admin

IMPORTANT: Remount affected clients to force new session negotiation at the restored value.

Step 3: Revert OpenShift MachineConfigs

Mirror of applying — delete the MachineConfig(s); each pool re-renders without the drop-in and rolls out another serial reboot, reverting params to defaults. One pool at a time, master last.

# worker first
oc delete machineconfig 99-worker-nfs-tuning
oc get mcp worker -w        # wait for UPDATED=True again

# then master (if applied)
oc delete machineconfig 99-master-nfs-tuning
oc get mcp master -w

# verify revert on a node (file gone, values default)
oc debug node/<node> -- chroot /host sh -c \
  'ls /etc/modprobe.d/nfs-tuning.conf 2>/dev/null && echo "FILE STILL PRESENT" || echo "file removed"; \
   cat /sys/module/sunrpc/parameters/tcp_max_slot_table_entries; \
   cat /sys/module/nfs/parameters/max_session_slots'
# expect: file removed, 65536, 64

Notes:

Each delete triggers another full rolling reboot of that pool — budget for it the same as the apply.
The MCD removes the drop-in as part of reconciling. If a node still shows the file after Done, that node didn't finish the revert — read its MCD log; don't hand-edit RHCOS (causes drift / Degraded).
Live values revert on reboot regardless, since modules reload without the options lines.

(Optional) Enable the node-exporter `mountstats` Collector in OpenShift

The mountstats collector gathers client-side NFS I/O statistics from each node's /proc/.../mountstats. It is disabled by default. Enabling it exposes metrics such as:

node_mountstats_nfs_read_bytes_total
node_mountstats_nfs_write_bytes_total
node_mountstats_nfs_operations_requests_total

Step 1 — Check whether the monitoring ConfigMap exists

oc -n openshift-monitoring get configmap cluster-monitoring-config

Step 2a — If it does NOT exist, create it

cat <<EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    nodeExporter:
      collectors:
        mountstats:
          enabled: true
EOF

Step 2b — If it DOES exist, edit it instead

oc -n openshift-monitoring edit configmap cluster-monitoring-config

Add the following under the existing config.yaml key (merge with any existing nodeExporter or collectors blocks):

data:
  config.yaml: |
    nodeExporter:
      collectors:
        mountstats:
          enabled: true

Step 3 — Wait for the Cluster Monitoring Operator to reconcile

The operator automatically rolls the node-exporter DaemonSet:

oc -n openshift-monitoring rollout status ds/node-exporter

Step 4 — Verify

Confirm the collector flag was added to the DaemonSet:

oc -n openshift-monitoring get ds node-exporter -o yaml | grep mountstats
# Expected output includes: --collector.mountstats

Confirm metrics are flowing. In the OpenShift web console go to Observe → Metrics and run:

node_mountstats_nfs_read_bytes_total

Or query via CLI:

TOKEN=$(oc whoami -t)
HOST=$(oc -n openshift-monitoring get route thanos-querier -o jsonpath='{.spec.host}')
curl -sk -H "Authorization: Bearer $TOKEN" \
  "https://$HOST/api/v1/query?query=node_mountstats_nfs_read_bytes_total" | jq .

Root Cause

Default RHEL/RHCOS NFS timeout settings are conservative and may not be optimized for high-throughput VM workloads on latent storage.

SBR

Virtualization

Product(s)

Red Hat OpenShift Container Platform

Components

storage

Category

Performance tune

Tags

nfs
storage

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Suggested NFS mount optimizations with NetApp Trident storage on OpenShift

Environment

Issue

Resolution

Prerequisites

Step 1: Update Trident Backend Configuration

Live Migrate or Start/Stop VMs

Step 2: NFS Client Tuning on OpenShift — MachineConfig (worker + master)

What this sets

Prerequisites

Apply — worker

Apply — master

Changing the values

Monitor the rollout

Validate on a node

Step 3: ONTAP NFSv4.1+ session slots: Incremental tuning guide

Command syntax

Incremental rollout plan

Rollback

Step 1: Revert Trident Backend Configuration

Step 2: Revert ONTAP NFSv4.1+ session slots

Step 3: Revert OpenShift MachineConfigs

(Optional) Enable the node-exporter mountstats Collector in OpenShift

Step 1 — Check whether the monitoring ConfigMap exists

Step 2a — If it does NOT exist, create it

Step 2b — If it DOES exist, edit it instead

Step 3 — Wait for the Cluster Monitoring Operator to reconcile

Step 4 — Verify

Root Cause

(Optional) Enable the node-exporter `mountstats` Collector in OpenShift