Initial baseline data collection for NFS client streaming I/O performance

Updated

The purpose of this article to provide a starting point when it is believed that NFS Client performance is slow or would like to be measured. The steps below provide useful baseline data from which to begin troubleshooting RHEL NFS Client speed, instead of spending time doing irrelevant tests like small cached dd or scp.

Network Tests

Test the speed of the underlying network

Test between the NFS Client subnet and the NFS Server if possible. If the NFS Server is a storage appliance which cannot run bandwidth testing utilities, use a system in the NFS Server's network subnet.

  • iperf
    Server: iperf3 -i 10 -s
    Client: iperf3 -i 10 -w 4M -t 60 -c <server hostname or ip address>

  • ttcp
    Server: ttcp -l 1048576 -b 4194304 -r -s
    Client: dd if=/dev/zero bs=1M count=10240 | ttcp -l 1048576 -b 4194304 -t -n 10240 <server hostname or ip address>

I/O tests

  • As these tests drop filesystem cache, they are not suitable for production usage.
  • Quiesce (make the system quiet) as much as possible by stopping running applications and preventing user activity before running the tests.

Data Collection During Each Test

  • cat /etc/fstab > /tmp/etc-fstab.out
  • cat /proc/mounts > /tmp/proc-mounts.out
  • nfsiostat 5 /mnt/nfs > /tmp/nfsiostat-test-type.out
  • iostat -x 5 > /tmp/iostat-test-type.out
  • while true; do date >> /tmp/nfs_meminfo-test-type.out; egrep "(Dirty|Writeback|NFS):" /proc/meminfo >> /tmp/nfs_meminfo-test-type.out; sleep 5; done
  • Replace the test-type in the above commands with local-read or local-write or nfs-write or nfs-read

Test the streaming write performance of the local storage

  • Drop cache before every test: echo 3 > /proc/sys/vm/drop_caches
  • dd if=/dev/zero of=/tmp/testfile.bin bs=1M count=10000 conv=fsync

Test the streaming read performance of the local storage

  • Drop cache before every test: echo 3 > /proc/sys/vm/drop_caches
  • dd if=/tmp/testfile.bin of=/dev/null bs=1M

Test the streaming write performance of the NFS client

  • Drop cache before every test: echo 3 > /proc/sys/vm/drop_caches
  • dd if=/dev/zero of=/mnt/nfs/testfile.bin bs=1M count=10000 conv=fsync

Test the streaming read performance of NFS client

  • Drop cache before every test: echo 3 > /proc/sys/vm/drop_caches
  • dd if=/mnt/nfs/testfile.bin of=/dev/null bs=1M

Assumptions

  • Everything is untuned and in the default configuration. Default socket buffers, default mount options, default network configuration.

Theory Points

  • Pick one NFS Client, one NFS Server, one NFS mount. We need consistency in our tests and results.
  • A large file transfer will only go as fast as the slowest point in the system. If the disk performs at 10Mb/sec and the network performs at 100Gb/sec, the file transfer will go at 10Mb/sec.
  • There is no point doing tests to the filesystem cache, such as dd without conv=fsync. Cached I/O is more a test of memory bandwidth and page flush settings than of disk performance.
  • Use a large block size (eg: 1MiB) and large repeat count (eg: 10000) to get a big streaming IO (eg: 10GiB) to fully exercise the storage.

Things Not To Use

  • dd without conv=fsync or oflag=direct. This uses filesystem cache.
  • scp. This is bottlenecked by a single CPU core performing encryption, and eventually by OpenSSH's built-in 64KiB buffer size.
  • rsync. This uses filesystem cache, and cannot bypass the cache.

Further Performance Tuning

Further performance tuning is outside the scope of this solution, however, the following are good ideas:

  • Make a test matrix and recording results of each test. eg:

    .--------------------------------- ------------ -----------.
| Client kernel                   | Write      | Read      |
+--------------------------------- ------------ -----------
| 2.6.32-220                      | 100Mb/sec  | 100Mb/sec |
| 2.6.32-279**                    | 110Mb/sec  | 110Mb/sec |
'--------------------------------- ------------ -----------'

changed parameter denoted with **

  • If you change any parameters, record how that parameter was set for all data points. eg:

    .---------------- -------- ------- ------------ -----------.
| Client kernel  | rsize  | wsize | Write      | Read      |
+---------------- -------- ------- ------------ -----------+
| 2.6.32-220     | 128k   | 128k  | 100Mb/sec  | 100Mb/sec |
| 2.6.32-279**   | 128k   | 128k  | 110Mb/sec  | 110Mb/sec |
| 2.6.32-279     | 1Mb**  | 1Mb** | 120Mb/sec  | 120Mb/sec |
'---------------- ---------------- ------------ -----------'

changed parameter denoted with **

  • Only change one parameter at a time. Record results of all parameter changes.

Product(s)
Category
Components
Tags
Article Type